Computer Science ›› 2023, Vol. 50 ›› Issue (7): 143-151.doi: 10.11896/jsjkx.220700232

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Medical Ultrasound Image Super-resolution Reconstruction Based on Video Multi-frame Fusion

ZHAO Ran, YUAN Jiabin, FAN Lili   

  1. School of Computer Science and Technology,Nanjing University of Aeronautics and Astronautics,Nanjing 211106,China
  • Received:2022-07-23 Revised:2022-11-22 Online:2023-07-15 Published:2023-07-05
  • About author:ZHAO Ran,born in 1998,postgraduate.Her main research interests include deep learning and medical image processing.YUAN Jiabin,born in 1968,Ph.D,professor,doctoral supervisor,is a member ofChina Computer Federation.His main research interests include high-performance computing,quantum computing,deep learning,medical image processing,etc.
  • Supported by:
    National Key Research and Development Program of China(2017YFB0802303) and National Natural Science Foundation of China(62076127).

Abstract: Medical ultrasound imaging is one of the most widely used methods in clinical diagnosis.However,the resolution and contrast of ultrasound images are low,and the noise is serious.Image Super-resolution is widely used to improve the quality of ultrasound images.However,the exsisting studies lack full use of complementary information in ultrasound video frames,so the results are not ideal.To solve this problem,this paper proposes a super-resolution method of medical ultrasound images based on video multi-frame fusion.Firstly,it builds a multi-frame fusion model based on convolutional neural network for ultrasound images.The model obtains the fused image with rich information by feature fusion of continuous multi-frame images.Then,it builds a lightweight image super-resolution model based on data-free knowledge distillation.The model trains fused feature images to obtain a teacher super-resolution model.To obtain the final high-quality medical ultrasound image,it creates a lightweight student super-resolution model using the trained teacher model and training data from the generative adversarial network.Finally,the proposed method is tested on large ultrasound datasets,using two objective image evaluation indicators and results on image classification to evaluate its performance.The results show that compared with other methods,the proposed method not only improves the resolution of ultrasound images,but also obtains ultrasound images with more information and higher contrast.The recognition accuracy of the super-resolution image obtained by this method in the classification network can reach 97.30%,which is much higher than that of previous approaches and can increase productivity and the precision of clinical diagnoses.

Key words: Medical ultrasound image, Multi-frame fusion, Super-resolution reconstruction, Data-free knowledge distillation, Convolutional neural network

CLC Number: 

  • TP391
[1]YAO Z W,YANG F,HUANG J,et al.Improved CycleGANs for Intravascular Ultrasound Image Enhancemen [J].Computer Science,2019,46(5):221-227.
[2]LIU H,LIU J,HOU S,et al.Perception consistency ultrasound image super-resolution via self-supervised CycleGAN [J].Neural Computing and Applications,2021,33(1):1-11.
[3]KUMAR P,SRIVASTAVA S,SAI Y P.High-PerformanceMedical Image Processing[M].New York:Apple Academic Press,2022:51-61.
[4]YU Z P,BAI G Z,LIU H Z,et al.Research on Underwater Image Enhancement Based on Histogram Stretching in UCM Algorithm[J].Journal of Chongqing Technology and Business University:Natural Science Edition,2022,39(5):10-16.
[5]BASHIR S M A,WANG Y,KHAN M,et al.A comprehensive review of deep learning-based single image super-resolution [J].PeerJ Computer Science,2021,7(7):621-677.
[6]KIM J S.Improved image resolution during zooming in ultra-sound image using deep learning technique[C]//2020 IEEE International Ultrasonics Symposium(IUS).IEEE Press,2020:1-3.
[7]SAWATN A,KULKARNI S.Ultrasound Image Enhancement using Super Resolution [J].Biomedical Engineering Advances,2022,3(1):1-9.
[8]GAO G,LI W,LI J,et al.Feature distillation interaction weighting network for lightweight image super-resolution[C]//Proceedings of the AAAI Conference on Artificial Intelligence.AAAI Press,2022:661-669.
[9]WANG X,XIE L,DONG C,et al.Real-esrgan:Training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE Press,2021:1905-1914.
[10]LUO Z,HUANG Y,LI S,et al.Learning the degradation distribution for blind image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE Press,2022:6063-6072.
[11]XIA B,HANG Y,TIAN Y,et al.Efficient Non-Local Contrastive Attention for Image Super-Resolution[C]//Proceedings of the AAAI Conference on Artificial Intelligence.AAAI Press,2022:6063-6072.
[12]LI Z,LI S,WANG J M,et al.A Novel Multi-Frame Color Images Super-Resolution Framework based on Deep Convolutional Neural Network[C]//Proceedings of the 2016 5th International Conference on Measurement,Instrumentation and Automation.Atlantis Press,2016.
[13]VANMALI A V,KATARIA T,KELKAR S G,et al.Ringing artifacts in wavelet based image fusion:Analysis,measurement and remedies [J].Information Fusion,2020,56(4):39-69.
[14]KAUR H,KOUNDAL D,KADYAN V.Image fusion tech-niques:a survey [J].Archives of computational methods in Engineering,2021,28(7):4425-4447.
[15]NOOR A,GAFFAR S,HASSAN M T,et al.Hybrid Image Fusion Method Based 0n Discrete Wavelet Transform(DWT),Principal Component Analysis(PCA) and Guided Filter [C]//2020 First International Conference of Smart Systems and Emerging Technologies.IEEE Press,2020:138-143.
[16]ABBASI AGHAMALEKI J,GHORBANI A.Infrared and visible image fusion based on optimal segmenting and contour extraction [J].SN Applied Sciences,2021,3(3):1-14.
[17]PAN Z,YU M,JIANG G,et al.Multi-exposure high dynamic range imaging with informative content enhanced network [J].Neurocomputing,2020,386(14):147-164.
[18]LI J,GUO X,LU G,et al.DRPL:Deep regression pair learning for multi-focus image fusion [J].IEEE Transactions on Image Processing,2020,29(3):4816-4831.
[19]MA B,ZHU Y,YIN X,et al.SESF-Fuse:an unsupervised deep model for multi-focus image fusion [J].Neural Computing and Applications,2021,33(11):5793-5804.
[20]MA J,TANG L,XU M,et al.STDFusionNet:An infrared and visible image fusion network based on salient target detection [J].IEEE Transactions on Instrumentation and Measurement,2021,70(4):1-13.
[21]GOU J P,YU B S,MAYBANK S J.Knowledge distillation:A survey [J].International Journal of Computer Vision,2021,129(6):1789-1819.
[22]MA K,ZENG K,WANG Z.Perceptual quality assessment for multi-exposure image fusion [J].IEEE Transactions on Image Processing,2015,24(11):3345-3356.
[23]ZHANG Y,LIU Y,SUN P,et al.IFCNN:A general image fusion framework based on convolutional neural network [J].Information Fusion,2020,54(2):99-118.
[24]KIM J,LEE J K,LEE K M.Accurate image super-resolutionusing very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE Press,2016:1646-1654.
[25]ZHANG Y,CHEN H,CHEN X H,et al.Data-free knowledge distillation for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Computer Vision Foundation,2021:7852-7861.
[26]LIM B,SON S,KIM H,et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu:IEEE Computer Society,2017:1132-1140.
[27]CHEN H,WANG Y H,XU C,et al.Data-free learning ofstudent networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.Seoul:IEEE Press,2019:3513-3521.
[28]TACO G.Ultrasound Images & Clips[EB/OL].www.ultrasoundcases.info.
[29]DONG C,LOY C C,TANG X.Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision.Amsterdam:Springer Press,2016:391-407.
[30]ZHANG Y,LI K,LI K,et al.Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision.Munich:Springer Press,2018:286-301.
[31]SHI W,ABALLERO J,HUSZAR F,et al.Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE Press,2016:1874-1883.
[1] XIONG Haojie, WEI Yi. Study on Multibeam Sonar Elevation Data Prediction Based on Improved CNN-BP [J]. Computer Science, 2023, 50(6A): 220100161-4.
[2] HUANG Yujiao, CHEN Mingkai, ZHENG Yuan, FAN Xinggang, XIAO Jie, LONG Haixia. Text Classification Based on Weakened Graph Convolutional Networks [J]. Computer Science, 2023, 50(6A): 220700039-5.
[3] LUO Ruiqi, YAN Jinlin, HU Xinrong, DING Lei. EEG Emotion Recognition Based on Multiple Directed Weighted Graph and ConvolutionalNeural Network [J]. Computer Science, 2023, 50(6A): 220600128-8.
[4] LI Han, HOU Shoulu, TONG Qiang, CHEN Tongtong, YANG Qimin, LIU Xiulei. Entity Relation Extraction Method in Weapon Field Based on DCNN and GLU [J]. Computer Science, 2023, 50(6A): 220200112-7.
[5] XU Changqian, WANG Dong, SU Feng, ZHANG Jun, BIAN Haifeng, LI Long. Image Recognition Method of Transmission Line Safety Risk Assessment Based on MultidimensionalData Coupling [J]. Computer Science, 2023, 50(6A): 220500032-6.
[6] LUO Huilan, LONG Jun, LIANG Miaomiao. Attentional Feature Fusion Approach for Siamese Network Based Object Tracking [J]. Computer Science, 2023, 50(6A): 220300237-9.
[7] WANG Jinwei, ZENG Kehui, ZHANG Jiawei, LUO Xiangyang, MA Bin. GAN-generated Face Detection Based on Space-Frequency Convolutional Neural Network [J]. Computer Science, 2023, 50(6): 216-224.
[8] ZHANG Xue, ZHAO Hui. Sentiment Analysis Based on Multi-event Semantic Enhancement [J]. Computer Science, 2023, 50(5): 238-247.
[9] WANG Lin, MENG Zuqiang, YANG Lina. Chinese Sentiment Analysis Based on CNN-BiLSTM Model of Multi-level and Multi-scale Feature Extraction [J]. Computer Science, 2023, 50(5): 248-254.
[10] YE Han, LI Xin, SUN Haichun. Convolutional Network Entity Missing Detection Method Combined with Gated Mechanism [J]. Computer Science, 2023, 50(5): 262-269.
[11] CHANG Liwei, LIU Xiujuan, QIAN Yuhua, GENG Haijun, LAI Yuping. Multi-source Fusion Network Security Situation Awareness Model Based on Convolutional Neural Network [J]. Computer Science, 2023, 50(5): 382-389.
[12] CAO Chenyang, YANG Xiaodong, DUAN Pengsong. WiDoor:Close-range Contactless Human Identification Approach [J]. Computer Science, 2023, 50(4): 388-396.
[13] SHAO Yunfei, SONG You, WANG Baohui. Study on Degree of Node Based Personalized Propagation of Neural Predictions forSocial Networks [J]. Computer Science, 2023, 50(4): 16-21.
[14] WANG Xiaofei, FAN Xueqiang, LI Zhangwei. Improving RNA Base Interactions Prediction Based on Transfer Learning and Multi-view Feature Fusion [J]. Computer Science, 2023, 50(3): 164-172.
[15] MEI Pengcheng, YANG Jibin, ZHANG Qiang, HUANG Xiang. Sound Event Joint Estimation Method Based on Three-dimension Convolution [J]. Computer Science, 2023, 50(3): 191-198.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!