基于深度学习的图像去模糊方法研究进展

doi:10.11896/jsjkx.201200043

摘要/Abstract

摘要： 近年来,随着便携、轻巧的数码成像设备的日益普及,人们获取图像的手段日益方便与灵活,数字图像在视频监控、医疗诊断、太空探测等领域起到了重要的作用。然而,在现有的成像过程中存在诸多问题,如相机的感光单元质量差、摄影者专业水平低、拍摄环境恶劣等,往往导致最终得到的图像含有明显的模糊以及噪声。如何使计算机自动地从模糊图像中把清晰的图像恢复出来,从而为其他的图像处理问题以及后续的计算机智能化分析提供高质量的图像,成为了亟待解决的问题。图像去模糊是典型的病态问题,解决该问题的常用方法主要包括基于统计先验建模和数据驱动的方法。然而,传统的统计先验建模的方法对清晰图像特征的刻画能力有限。而数据驱动的方法尤其是以深度学习为代表的方法依靠其强大的特征表示能力,为解决图像去模糊提供了一种新的、有效的方式。文中基于深度学习的图像去模糊方法,概述了目前图像去模糊方法的研究现状,分析了当前方法所面临的问题,并展望了图像去模糊方法的研究趋势。

关键词: 病态问题, 深度学习, 图像去模糊, 先验建模, 运动模糊

Abstract: With the increasing development of portable and smart digital imaging devices,the way to capture photos is more convenient and flexible.Digital images play an important role in video surveillance,medical diagnosis,space exploration,and so on.However,the captured images usually contain significant blur and noise due to the limited quality of the camera sensors,the skill of the photographers,and the imaging environments.How to restore the clear images from blurry ones so that they can facilitate the following intelligent analysis tasks is important but challenging.Image deburring is a classical ill-posed problem.Represented methods for this problem include the statistical prior-based methods and data-driven methods.However,conventional statistical prior-based methods have limited ability for modeling the inherent properties of the clear images.The data-driven methods,especially the deep learning methods,provide an effective way to solve image deblurring.This paper focuses on the deep learning-based image deblurring methods.It first introduces the research progress of the image deblurring problem,and then analyzes the challenges of the image deblurring problem.Finally,it discusses the research trends of the image deblurring problem.

Key words: Deep learning, Ill-posed problem, Image deblurring, Motion blur, Prior modeling

中图分类号:

TP391

潘金山. 基于深度学习的图像去模糊方法研究进展[J]. 计算机科学, 2021, 48(3): 9-13. https://doi.org/10.11896/jsjkx.201200043

PAN Jin-shan. Research Progress on Deep Learning-based Image Deblurring[J]. Computer Science, 2021, 48(3): 9-13. https://doi.org/10.11896/jsjkx.201200043

参考文献

[1]FERGUS R,SINGH B,HERTZMANN A,et al.Freeman,Re-moving camera shake from a single photograph [J].ACM Transactions on Graphics,2006,25(3):787-794.
[2]KRISHNAN D,TAY T,FERGUS R.Blind deconvolution using a normalized sparsity measure [C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2011:233-240.
[3]LEVIN A,WEISS Y,DURAND F,et al.Efficient marginal likelihood optimization in blind deconvolution[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition (CVPR).2011:2657-2664.
[4]XU L,ZHENG S C,JIA J Y.Unnatural L0 sparse representation for natural image deblurring[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2013:1107-1114.
[5]SHAN Q,JIA J Y,AGARWALA A.High-quality motion de-blurring from a single image[J].ACM Transactions on Gra-phics,2008,27(3):73.
[6]MICHAELI T,IRANI M.Blind DEBLURRING Using Internal Patch Recurrence[C]//European Conference on Computer Vision.2014:783-798.
[7]CHEN X G,HE X J,YANG J,et al.An effective document ima-ge deblurring algorithm [C]//IEEE Conference on Computer Vision and Pattern Recognition.2011:369-376.
[8]CHO H,WANG J,LEE S.Text image deblurring using text-specific properties[C]//European Conference on Computer Vision.2012:524-537.
[9]PAN J S,HU Z,SU Z X,et al.Deblurring text images via L0-regularized intensity and gradient prior[C]//IEEE Conference on Computer Vision and Pattern Recognition.2014:2901-2908.
[10]PAN J S,HU Z,SU Z X,et al.Deblurring Face Images with Exemplars[C]//European Conference on Computer Vision.2014:47-62.
[11]PAN J S,SUN D Q,PFISTER H,et al.Deblurring images via dark channel prior [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,40(10):2315-2328.
[12]ZHANG J W,PAN J S,LAI W S,et al.Learning fully convolutional networks for iterative non-blind deconvolution[C]//IEEE Conference on Computer Vision and Pattern Recognition.2017:6969-6977.
[13]ZHANG K,ZUO W M,GU S H,et al.Learning deep CNN denoiser prior for image restoration[C]//IEEE Conference on Computer Vision and Pattern Recognition.2017:2808-2817.
[14]LI L,PAN JS,LAI W S,et al.Learning a Discriminative Prior for Blind Image Deblurring[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:6616-6625.
[15]LEVIN A,WEISS Y,DURAND F,et al.Understanding andevaluating blind deconvolution algorithms[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.2009:1964-1971.
[16]KOHLER R,HIRSCH M,MOHLER B,et al.Stefan Harme-ling:Recording and playback of camera shake:benchmarking blind deconvolution with a real-world database[C]//European Conference on Computer Vision.2012:27-40.
[17]SUN J,CAO W F,XU Z B,et al.Learning a convolutional neural network for non-uniform motion blur removal[C]//IEEE Conference on Computer Vision and Pattern Recognition.2015:769-777.
[18]ZORAN D,WEISS Y.From learning models of natural image patches to whole image restoration[C]//IEEE International Conference on Computer Vision.2011:479-486.
[19]GONG D,YANG J,LIU L Q,et al.From motion blur to motion flow:a deep learning solution for removing heterogeneous motion blur[C]//IEEE Conference on Computer Vision and Pattern Recognition.2017:3806-3815.
[20]SCHULER C J,HIRSCH M,HARMELING S,et al.Learning to deblur [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(7):1439-1451.
[21]PAN J S,REN W Q,HU Z,et al.Learning to Deblur Images with Exemplars [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(6):1412-1425.
[22]XU L,LU C W,XU Y,et al.Image smoothing via L0 gradient minimization.photograph [J].ACM Transactions on Graphics,2011,30(6):174.
[23]NAH S J,KIM T H,LEE K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//IEEE Conference on Computer Vision and Pattern Recognition.2017:257-265.
[24]HE K M,ZHANG X Y,REN S Q,et al.Deep Residual Learning for Image Recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[25]TAO X,GAO H Y,SHEN X Y,et al.Scale-recurrent network for deep image deblurring[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:8174-8182.
[26]ZHANG J W,PAN J S,REN J S J,et al.Dynamic Scene Deblurring Using Spatially Variant Recurrent Neural Networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:2521-2529.
[27]GAO H Y,TAO X,SHEN X Y,et al.Dynamic Scene Deblurring With Parameter Selective Sharing and Nested Skip Connections[C]//IEEE Conference on Computer Vision and Pattern Recognition.2019:3848-3856.
[28]ZHANG H G,DAI Y C,LI H D,et al.Deep Stacked Hierarchical Multi-Patch Network for Image Deblurring[C]//IEEE Conference on Computer Vision and Pattern Recognition.2019:5978-5986.
[29]ALJADAANY R,PAL D K,SAVVIDES M.Douglas-Rachford Networks:Learning Both the Image Prior and Data Fidelity Terms for Blind Image Deconvolution[C]//IEEE Conference on Computer Vision and Pattern Recognition.2019:10235-10244.
[30]SUIN M,PUROHIT K,RAJAGOPALAN A N.Spatially-At-tentive Patch-Hierarchical Network for Adaptive Motion Deblurring[C]//IEEE Conference on Computer Vision and Pattern Recognition.2020:3603-3612.
[31]YUAN Y,SU W,MA D D.Efficient Dynamic Scene Deblurring Using Spatially Variant Deconvolution Network With Optical Flow Guided Training[C]//IEEE Conference on Computer Vision and Pattern Recognition.2020:3552-3561.
[32]KUPYN O,BUDZAN V,MYKHAILYCH M,et al.Deblur-GAN:Blind motion deblurring using conditional adversarial networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:8183-8192.
[33]KUPYN O,MARTYNIUK T,WU J R,et al.DeblurGAN-v2:Deblurring (orders-of-magnitude) faster and better [C]//IEEE International Conference on Computer Vision.2019:8877-8886.
[34]NIMISHA T M,KUMAR S,RAJAGOPALAN A N.Unsupervised Class-Specific Deblurring[C]//European Conference on Computer Vision.2018:358-374.
[35]LU B Y,CHEN J C,CHELLAPPA R.Unsupervised Domain-Specific Deblurring via Disentangled Representations[C]//IEEE Conference on Computer Vision and Pattern Recognition.2019:10225-10234.
[36]ZHANG K H,LUO W H,ZHONG Y R,et al.Deblurring by Realistic Blurring[C]//IEEE Conference on Computer Vision and Pattern Recognition.2020:2734-2743.
[37]PAN J S,LIU Y,DONG J X,et al.Physics-Based Generative Adversarial Models for Image Restoration and Beyond.deblur [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020.
[38]LIN S N,ZHANG J W,PAN J S,et al.Learning to Deblur Face Images via Sketch Synthesis[C]//AAAI Conference on Artificial Intelligence.2020:11523-11530.
[39]SUN L B,CHO S,WANG J,et al.Edge-based blur kernel estimation using patch priors[C]//IEEE International Conference on Computational Photography.2013:1-8.
[40]LAI W S,HUANG J B,HU Z,et al.A comparative study for single image blind deblurring[C]//IEEE Conference on Computer Vision and Pattern Recognition.2016:1701-1709.
[41]SU S C,DELBRACIO M,WANG J,et al.Deep video deblurring for hand-held cameras[C]//IEEE Conference on Computer Vision and Pattern Recognition.2017:237-246.
[42]NAH S J,BAIK S Y,HONG S,et al.NTIRE 2019 challenge on video deblurring and super-resolution:Dataset and study[C]//IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:1996-2005.
[43]RIM J,LEE H,WON J,et al.Real-World Blur Dataset forLearning and Benchmarking Deblurring Algorithms[C]//European Conference on Computer Vision.2020:184-201.
[44]ASIM M,SHAMSHAD F,AHMED A.Blind Image Deconvolution using Pretrained Generative Priors[C]//British Machine Vision Conference.2019:205.
[45]REN D W,ZHANG K,WANG Q L,et al.Neural Blind Deconvolution Using Deep Priors [C]//IEEE Conference on Compu-ter Vision and Pattern Recognition.2020:3338-3347.
[46]PAN J S,BAI H R,TANG J H.Cascaded Deep Video Deblurring Using Temporal Sharpness Prior[C]//IEEE Conference on Computer Vision and Pattern Recognition.2020:3040-3048.
[47]ZHOU S C,ZHANG J W,PAN J S,et al.Spatio-temporal filter adaptive network for video deblurring[C]//IEEE International Conference on Computer Vision.2019:2482-2491.
[48]NAH S J,SON S Y,LEE K M.Recurrent neural networks with intra-frame iterations for video deblurring[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.2019:8102-8111.
[49]KIM T H,LEE K M,SCHOLKOPF B,et al.Online video deblurring via dynamic temporal blending network[C]//IEEE International Conference on Computer Vision.2017:4058-4067.
[50]KIM T H,SAJJADI M S M,HIRSCH M,et al.Spatio-temporal transformer network for video restoration[C]//European Conference on Computer Vision.2018:111-127.
[51]JIANG Z,ZHANG Y,ZOU D Q,et al.Learning Event-BasedMotion Deblurring[C]//IEEE Conference on Computer Vision and Pattern Recognition.2020:3317-3326.
[52]LIN S N,ZHANG J W,PAN J S,et al.Learning Event-Driven Video Deblurring and Interpolation[C]//European Conference on Computer Vision.2020:695-710.

相关文章 15

[1]	徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[2]	饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[3]	汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[4]	王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[5]	郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[6]	姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[7]	孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[8]	胡艳羽, 赵龙, 董祥军. 一种用于癌症分类的两阶段深度特征选择提取算法 Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification 计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[9]	程成, 降爱莲. 基于多路径特征提取的实时语义分割方法 Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction 计算机科学, 2022, 49(7): 120-126. https://doi.org/10.11896/jsjkx.210500157
[10]	侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木. 中文预训练模型研究进展 Advances in Chinese Pre-training Models 计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[11]	周慧, 施皓晨, 屠要峰, 黄圣君. 基于主动采样的深度鲁棒神经网络学习 Robust Deep Neural Network Learning Based on Active Sampling 计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[12]	苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫. 小样本雷达辐射源识别的深度学习方法综述 Survey of Deep Learning for Radar Emitter Identification Based on Small Sample 计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138
[13]	王君锋, 刘凡, 杨赛, 吕坦悦, 陈峙宇, 许峰. 基于多源迁移学习的大坝裂缝检测 Dam Crack Detection Based on Multi-source Transfer Learning 计算机科学, 2022, 49(6A): 319-324. https://doi.org/10.11896/jsjkx.210500124
[14]	楚玉春, 龚航, 王学芳, 刘培顺. 基于YOLOv4的目标检测知识蒸馏算法研究 Study on Knowledge Distillation of Target Detection Algorithm Based on YOLOv4 计算机科学, 2022, 49(6A): 337-344. https://doi.org/10.11896/jsjkx.210600204
[15]	刘伟业, 鲁慧民, 李玉鹏, 马宁. 指静脉识别技术研究综述 Survey on Finger Vein Recognition Research 计算机科学, 2022, 49(6A): 1-11. https://doi.org/10.11896/jsjkx.210400056

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed