基于深度学习的图像补全算法综述

doi:10.11896/jsjkx.200600009

摘要/Abstract

摘要： 图像补全是图像处理的一个研究领域,为有物体遮挡以及图像关键部分缺失状况下的图像识别提供了解决方案,应用领域非常广泛,受到了人们的关注。经深度学习方法补全的图像具有更高的图像分辨率和可靠性,逐渐成为图像补全的主流方法之一。文中针对图像补全领域的主要问题,介绍了相关深度学习方法的基本原理和经典算法,系统而渐进地剖析了2010年以来有代表性的图像补全方法,探讨了基于深度学习的图像补全在不同领域的具体应用,并列举了该研究领域目前面临的几个问题。

关键词: 上下文编码, 深度学习, 生成对抗网络, 图像补全

Abstract: Image inpainting is a research field of image processing that provides solutions for image recognition in the presence of object occlusion and in the absence of critical parts of the image,attracts widespread attention in a wide range of fields.Image inpainted by deep learning methods have higher image resolution and reliability,which makes deep learning one of the mainstream methods of image inpainting.This paper introduces the basic principles and classical algorithms of the relevant deep learning methods,systematically and progressively dissects the representative image inpainting methods since 2010,explores the specific applications of deep learning-based image inpainting in different fields,and lists several research problems faced by this research field currently.

Key words: Context encoder, Deep learning, Generative adversarial networks, Image inpainting

中图分类号:

TP391

唐浩丰, 董元方, 张依桐, 孙娟娟. 基于深度学习的图像补全算法综述[J]. 计算机科学, 2020, 47(11A): 151-164. https://doi.org/10.11896/jsjkx.200600009

TANG Hao-feng, DONG Yuan-fang, ZHANG Yi-tong, SUN Juan-juan. Survey of Image Inpainting Algorithms Based on Deep Learning[J]. Computer Science, 2020, 47(11A): 151-164. https://doi.org/10.11896/jsjkx.200600009

参考文献

[1] BERTALMIO,SAPIRO G.Image inpainting [C]//Proceedings of International Conference on Computer Graphics an Interactive Techniques.New Orleans,Louisiana,USA,2000:417-424.
[2] CRIMINISI A,PEREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[3] HINTON G E.A Practical Guide to Training Restricted Boltzmann Machines[J].Momentum,2012,9(1):599-619.
[4] BENGIO Y.Learning Deep Architectures for AI[J].Founda-tions and Trends in Machine Learning,2009,2(1):1-127.
[5] LIU G,REDA F A,SHIH K,et al.Image inpainting for irregular holes using partial convolutions[C]//European Conference on Computer Vision (ECCV) 2018.Computer Vision,2018(15):89-105.
[6] PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2016:2536-2544.
[7] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//International Conference on Neural Information Processing Systems.2014.
[8] RUMMELHART D E,HINTON G E,WILLIAMS R J.Learning Internal Representations by Error Propagation[J].Readings in Cognitive Science,1986,323(2):318-362.
[9] HINTON G E.Reducing the Dimensionality of Data with Neural Networks[J].Science,2006,313(5786):504-507.
[10] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet Classification with Deep Convolutional Neural Networks[C]//NIPS,2012.
[11] RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,Cham,2015.
[12] YU J H,LIN Z,YANG J M,et al.Free-Form Image Inpainting with Gated Convolution[C]//ARXIV.2018.
[13] YANG C,LU X,LIN Z,et al.High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis[C]//CVPR.2017:6721-6729.
[14] HONG X,XIONG P F,JI R H,et al.Deep Fusion Network for Image Completion[J].arXiv 1904.08060,2019.
[15] NAZERI K,NG E,JOSEPH T,et al.EdgeConnect:Generative Image Inpainting with AdversarialEdgeLearning[J].arXiv1901.00212,2019.
[16] YEH R A,CHEN C,LIM T Y,et al.Image Inpainting withDeep Generative Models[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).New York:IEEE Press,2017:5485-5489.
[17] RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[C]//Proc of International Conference on Learning Representations.2016:1-16.
[18] IIZUKA S,SIMO-SERRA E,ISHIKAWA H .Globally and locally consistent image completion[J].ACM Transactions on Graphics,2017,36(4):1-14.
[19] DOLHANSKY B,FERRER C C,WAY H,et al.Eye In-Painting with Exemplar Generative Adversarial Networks[C]//CVPR.2018:00577.
[20] YU J H,LIN Z,YANG J M,et al.Generative Image Inpainting with Contextual Attention[C]//CVPR.2018.
[21] LI H F,LI G B,LIN L,et al.Context-Aware Semantic Inpainting[J].IEEE Transactions on Cybernetics (T-Cybernetics).DOI:10.1109/TCYB.2018.2865036,2019.
[22] ZHENG C X,CHAM T J,CAI J F.Pluralistic Image Completion[C]//CVPR.2019.
[23] CHEN K,QIAO Q,SONG Z J.Applications of Generative Adversarial Nets in Medical Image Processing[J].Life Science Instruments,2008,6:71-81.
[24] YANG G,YU S,DONG H,et al.DAGAN:Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction[J].IEEE Transactions on Medical Imaging,2018,37(6):1310-1321.
[25] SHITRIT O,RIKLIN RAVIV T.Accelerated Magnetic Reso-nance Imaging by Adversarial Neural Network[C]//Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support,2017:30-38.
[26] QUAN T M,NGUYEN-DUC T,JEONG W.Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss[J].IEEE Transactions on Medical Imaging,2018,37(6):1488-1497.
[27] BI L,KIM J,KUMAR A,et al.Synthesis of Positron Emission Tomography (PET) Images via Multi-channel Generative Adversarial Networks (GANs)[C]//Molecular Imaging,Recons-truction and Analysis of Moving Body Organs,and Stroke Imaging and Treatment.2017:43-51.
[28] WANG Y,YU B,WANG L,et al.3D conditional generative adversarial networks for high-quality PET image estimation at low dose[J].NeuroImage,2018,174:550-562.
[29] DENG Y,LOY C C,TANG X O.Aesthetic-driven Image En-hancement by Adversarial Learning[C]//2018 ACM Multimedia Conference.ACM,Amsterdam,Seoul,South Korea.New York:ACM,2018:870-878.
[30] PERRONNIN F.AVA:A Large-scale Database for AestheticVisual Analysis[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition.RI,USA.New Jersey:IEEE,2012:2408-2415.
[31] LU Q W,TAO Q C,ZHAO Y L,et al.Simplified Based onComic Draft Drawings Generated Against the Web[J].Acta Automatica Sinica,2018,44(5):840-854.
[32] DONG C,LOY C C,HE K M,et al.Image superresolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-307.
[33] LIANG Y,WANG J J,ZHOU S P,et al.Incorporating imagepriors with deep convolutional neural networks for image super-resolution[J].Neurocomputing,2016,194:340-347.
[34] GU S H,ZUO W M,XIE Q,et al.Convolutional sparse coding for image super-resolution[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV).Santiago,Chile:IEEE,2015:1823-1831.
[35] YANG J C,WRIGHT J,HUANG T,et al.Image super-resolution as sparse representation of raw image patches[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition.Anchorage,Alaska,USA:IEEE,2008:1-8.
[36] TELEA A.An image inpainting technique based on the fastmarching method [J].Journal of Graphics Tools,2004,9(1):23-34.
[37] PETSCHNIGG G,SZELISKI R,AGRAWALA M,et al.Digital photography with flash and no-flash image pairs [J].ACM Trans on Graphics,2004,23(3):664-672.
[38] LAI K,BO L F,REN X F,et al.A large-scale hierar chical multi-view RGB-D object dataset [C]// 2011 IEEE International Conference on Robotics and Automation (ICRA).2011:1817-1824.
[39] ZHANG Y D,FUNKHOUSER T.Deep Depth Completion of a Single RGB-D Image[J].Computer Vision and Pattern Recognition (CVPR 2018),2018,3:175-185.
[40] LI S M,LEI G Q,FAN R.Depth Map Super-Resolution Based on Deep Convolutional Neural Networks[J].Acta Optica Sinica,2017,37(12):1210002.
[41] YU S X,HU L M,ZHANG J,et al.Depth image super-resolution reconstruction with two-channel pyramid convolutional neural networks[J].Application Research of Computers,2019,2:96.
[42] LEI Y M.The role of video surveillance images in detecting and solving crimes[J].Engineering Technology,2016(12):302-302.
[43] WANG X D,WEI H Q,GAO C,et al.Identity preserving face complexion with generative adversarial networks[J].Chinese Journal of Netword and Information Security,2018,4(8):71-76.
[44] DING Y D,YU B.Application Research of New Computer Vision Technology in Old Film Restoration[J].Advanced Motion Picture Technology,2018,8:8.
[45] XIA T R,YU Y D,YU B.An old film restoration method using subframe quilting[J].Journal of Shanghai University,2018,24(4):503-511.
[46] SATOSHI I,EDGAR S S,ISHIKAWA H.Let there be Color!:Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification[J].Acm Transactions on Graphics,2016,35(4):1-11.
[47] ZHU Z.Image Enhancement and Analysis for Street Views[D].Tsinghua University,2016.

相关文章 15

[1]	张佳, 董守斌. 基于评论方面级用户偏好迁移的跨领域推荐算法 Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer 计算机科学, 2022, 49(9): 41-47. https://doi.org/10.11896/jsjkx.220200131
[2]	徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[3]	饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[4]	汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[5]	王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[6]	郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[7]	姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[8]	孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[9]	胡艳羽, 赵龙, 董祥军. 一种用于癌症分类的两阶段深度特征选择提取算法 Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification 计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[10]	戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮. 基于DNGAN的磁共振图像超分辨率重建算法 Super-resolution Reconstruction of MRI Based on DNGAN 计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105
[11]	程成, 降爱莲. 基于多路径特征提取的实时语义分割方法 Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction 计算机科学, 2022, 49(7): 120-126. https://doi.org/10.11896/jsjkx.210500157
[12]	侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木. 中文预训练模型研究进展 Advances in Chinese Pre-training Models 计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[13]	周慧, 施皓晨, 屠要峰, 黄圣君. 基于主动采样的深度鲁棒神经网络学习 Robust Deep Neural Network Learning Based on Active Sampling 计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[14]	苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫. 小样本雷达辐射源识别的深度学习方法综述 Survey of Deep Learning for Radar Emitter Identification Based on Small Sample 计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138
[15]	王君锋, 刘凡, 杨赛, 吕坦悦, 陈峙宇, 许峰. 基于多源迁移学习的大坝裂缝检测 Dam Crack Detection Based on Multi-source Transfer Learning 计算机科学, 2022, 49(6A): 319-324. https://doi.org/10.11896/jsjkx.210500124

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed