计算机科学

计算机科学 ›› 2023, Vol. 50 ›› Issue (7): 143-151.doi: 10.11896/jsjkx.220700232

• 计算机图形学&多媒体 • 上一篇    下一篇

基于视频多帧融合的医学超声图像超分辨率重建方法

赵冉, 袁家斌, 范利利   

  1. 南京航空航天大学计算机科学与技术学院 南京 211106
  • 收稿日期:2022-07-23 修回日期:2022-11-22 出版日期:2023-07-15 发布日期:2023-07-05
  • 通讯作者: 袁家斌(jbyuan@nuaa.edu.cn)
  • 作者简介:(zhaoran@nuaa.edu.cn)
  • 基金资助:
    国家重点研发计划(2017YFB0802303);国家自然科学基金(62076127)

Medical Ultrasound Image Super-resolution Reconstruction Based on Video Multi-frame Fusion

ZHAO Ran, YUAN Jiabin, FAN Lili   

  1. School of Computer Science and Technology,Nanjing University of Aeronautics and Astronautics,Nanjing 211106,China
  • Received:2022-07-23 Revised:2022-11-22 Online:2023-07-15 Published:2023-07-05
  • About author:ZHAO Ran,born in 1998,postgraduate.Her main research interests include deep learning and medical image processing.YUAN Jiabin,born in 1968,Ph.D,professor,doctoral supervisor,is a member ofChina Computer Federation.His main research interests include high-performance computing,quantum computing,deep learning,medical image processing,etc.
  • Supported by:
    National Key Research and Development Program of China(2017YFB0802303) and National Natural Science Foundation of China(62076127).

PDF (PC)

728

摘要： 医学超声成像是临床诊断中应用最广泛的成像方式之一。目前超声图像普遍存在分辨率和对比度较低的问题,并且成像过程易受噪声污染。图像超分辨率重建技术被广泛用于改善超声图像的质量。然而,已有的研究工作缺乏对超声视频帧之间互补信息的充分利用,因此效果并不理想。针对此问题,提出了一种基于视频多帧融合的医学超声图像超分辨率重建方法。首先,构建了一个基于卷积神经网络的无监督多帧融合模型,该模型通过对连续的多帧图像进行特征融合,得到具有丰富信息的融合特征图像;然后,建立一个基于无数据知识蒸馏的轻量级图像超分辨率重建模型,通过训练融合特征图像得到教师超分辨率网络,利用训练好的教师网络和生成对抗网络获取的训练数据得到轻量级学生超分网络,最终得到高质量的医学超声图像;最后,在大型超声数据集上进行实验,采用两种图像客观评价指标以及图像分类任务进行评估。结果表明,所提方法与8种已有的图像超分辨率重建方法相比,在提高超声图像分辨率的同时,获得了包含更多信息且具有更高对比度的超声图像。此外,所提方法得到的超分辨率图像在分类网络的识别准确率可达到97.30%,明显优于其他方法,可提高临床诊断效率与准确性。

关键词: 医学超声图像, 多帧融合, 超分辨率重建, 无数据知识蒸馏, 卷积神经网络

Abstract: Medical ultrasound imaging is one of the most widely used methods in clinical diagnosis.However,the resolution and contrast of ultrasound images are low,and the noise is serious.Image Super-resolution is widely used to improve the quality of ultrasound images.However,the exsisting studies lack full use of complementary information in ultrasound video frames,so the results are not ideal.To solve this problem,this paper proposes a super-resolution method of medical ultrasound images based on video multi-frame fusion.Firstly,it builds a multi-frame fusion model based on convolutional neural network for ultrasound images.The model obtains the fused image with rich information by feature fusion of continuous multi-frame images.Then,it builds a lightweight image super-resolution model based on data-free knowledge distillation.The model trains fused feature images to obtain a teacher super-resolution model.To obtain the final high-quality medical ultrasound image,it creates a lightweight student super-resolution model using the trained teacher model and training data from the generative adversarial network.Finally,the proposed method is tested on large ultrasound datasets,using two objective image evaluation indicators and results on image classification to evaluate its performance.The results show that compared with other methods,the proposed method not only improves the resolution of ultrasound images,but also obtains ultrasound images with more information and higher contrast.The recognition accuracy of the super-resolution image obtained by this method in the classification network can reach 97.30%,which is much higher than that of previous approaches and can increase productivity and the precision of clinical diagnoses.

Key words: Medical ultrasound image, Multi-frame fusion, Super-resolution reconstruction, Data-free knowledge distillation, Convolutional neural network

中图分类号: 

  • TP391
[1]YAO Z W,YANG F,HUANG J,et al.Improved CycleGANs for Intravascular Ultrasound Image Enhancemen [J].Computer Science,2019,46(5):221-227.
[2]LIU H,LIU J,HOU S,et al.Perception consistency ultrasound image super-resolution via self-supervised CycleGAN [J].Neural Computing and Applications,2021,33(1):1-11.
[3]KUMAR P,SRIVASTAVA S,SAI Y P.High-PerformanceMedical Image Processing[M].New York:Apple Academic Press,2022:51-61.
[4]YU Z P,BAI G Z,LIU H Z,et al.Research on Underwater Image Enhancement Based on Histogram Stretching in UCM Algorithm[J].Journal of Chongqing Technology and Business University:Natural Science Edition,2022,39(5):10-16.
[5]BASHIR S M A,WANG Y,KHAN M,et al.A comprehensive review of deep learning-based single image super-resolution [J].PeerJ Computer Science,2021,7(7):621-677.
[6]KIM J S.Improved image resolution during zooming in ultra-sound image using deep learning technique[C]//2020 IEEE International Ultrasonics Symposium(IUS).IEEE Press,2020:1-3.
[7]SAWATN A,KULKARNI S.Ultrasound Image Enhancement using Super Resolution [J].Biomedical Engineering Advances,2022,3(1):1-9.
[8]GAO G,LI W,LI J,et al.Feature distillation interaction weighting network for lightweight image super-resolution[C]//Proceedings of the AAAI Conference on Artificial Intelligence.AAAI Press,2022:661-669.
[9]WANG X,XIE L,DONG C,et al.Real-esrgan:Training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE Press,2021:1905-1914.
[10]LUO Z,HUANG Y,LI S,et al.Learning the degradation distribution for blind image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE Press,2022:6063-6072.
[11]XIA B,HANG Y,TIAN Y,et al.Efficient Non-Local Contrastive Attention for Image Super-Resolution[C]//Proceedings of the AAAI Conference on Artificial Intelligence.AAAI Press,2022:6063-6072.
[12]LI Z,LI S,WANG J M,et al.A Novel Multi-Frame Color Images Super-Resolution Framework based on Deep Convolutional Neural Network[C]//Proceedings of the 2016 5th International Conference on Measurement,Instrumentation and Automation.Atlantis Press,2016.
[13]VANMALI A V,KATARIA T,KELKAR S G,et al.Ringing artifacts in wavelet based image fusion:Analysis,measurement and remedies [J].Information Fusion,2020,56(4):39-69.
[14]KAUR H,KOUNDAL D,KADYAN V.Image fusion tech-niques:a survey [J].Archives of computational methods in Engineering,2021,28(7):4425-4447.
[15]NOOR A,GAFFAR S,HASSAN M T,et al.Hybrid Image Fusion Method Based 0n Discrete Wavelet Transform(DWT),Principal Component Analysis(PCA) and Guided Filter [C]//2020 First International Conference of Smart Systems and Emerging Technologies.IEEE Press,2020:138-143.
[16]ABBASI AGHAMALEKI J,GHORBANI A.Infrared and visible image fusion based on optimal segmenting and contour extraction [J].SN Applied Sciences,2021,3(3):1-14.
[17]PAN Z,YU M,JIANG G,et al.Multi-exposure high dynamic range imaging with informative content enhanced network [J].Neurocomputing,2020,386(14):147-164.
[18]LI J,GUO X,LU G,et al.DRPL:Deep regression pair learning for multi-focus image fusion [J].IEEE Transactions on Image Processing,2020,29(3):4816-4831.
[19]MA B,ZHU Y,YIN X,et al.SESF-Fuse:an unsupervised deep model for multi-focus image fusion [J].Neural Computing and Applications,2021,33(11):5793-5804.
[20]MA J,TANG L,XU M,et al.STDFusionNet:An infrared and visible image fusion network based on salient target detection [J].IEEE Transactions on Instrumentation and Measurement,2021,70(4):1-13.
[21]GOU J P,YU B S,MAYBANK S J.Knowledge distillation:A survey [J].International Journal of Computer Vision,2021,129(6):1789-1819.
[22]MA K,ZENG K,WANG Z.Perceptual quality assessment for multi-exposure image fusion [J].IEEE Transactions on Image Processing,2015,24(11):3345-3356.
[23]ZHANG Y,LIU Y,SUN P,et al.IFCNN:A general image fusion framework based on convolutional neural network [J].Information Fusion,2020,54(2):99-118.
[24]KIM J,LEE J K,LEE K M.Accurate image super-resolutionusing very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE Press,2016:1646-1654.
[25]ZHANG Y,CHEN H,CHEN X H,et al.Data-free knowledge distillation for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Computer Vision Foundation,2021:7852-7861.
[26]LIM B,SON S,KIM H,et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu:IEEE Computer Society,2017:1132-1140.
[27]CHEN H,WANG Y H,XU C,et al.Data-free learning ofstudent networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.Seoul:IEEE Press,2019:3513-3521.
[28]TACO G.Ultrasound Images & Clips[EB/OL].www.ultrasoundcases.info.
[29]DONG C,LOY C C,TANG X.Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision.Amsterdam:Springer Press,2016:391-407.
[30]ZHANG Y,LI K,LI K,et al.Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision.Munich:Springer Press,2018:286-301.
[31]SHI W,ABALLERO J,HUSZAR F,et al.Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE Press,2016:1874-1883.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发