Computer Science ›› 2020, Vol. 47 ›› Issue (11A): 151-164.doi: 10.11896/jsjkx.200600009

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Survey of Image Inpainting Algorithms Based on Deep Learning

TANG Hao-feng, DONG Yuan-fang, ZHANG Yi-tong, SUN Juan-juan   

  1. School of Economic and Management,Changchun University of Science and Technology,Changchun 130022,China
  • Online:2020-11-15 Published:2020-11-17
  • About author:TANG Hao-feng,born in 1999,undergraduate.His main research interests include deep learning and image recognition.
    DONG Yuan-fang,born in 1975,Ph.D,associate professor,master supervisor,is a member of China Computer Federation.Her main research interest include machine learning and so on.
  • Supported by:
    This work was supported by the Natural Science Foundation of Jilin Province,China(20150101053JC).

Abstract: Image inpainting is a research field of image processing that provides solutions for image recognition in the presence of object occlusion and in the absence of critical parts of the image,attracts widespread attention in a wide range of fields.Image inpainted by deep learning methods have higher image resolution and reliability,which makes deep learning one of the mainstream methods of image inpainting.This paper introduces the basic principles and classical algorithms of the relevant deep learning methods,systematically and progressively dissects the representative image inpainting methods since 2010,explores the specific applications of deep learning-based image inpainting in different fields,and lists several research problems faced by this research field currently.

Key words: Context encoder, Deep learning, Generative adversarial networks, Image inpainting

CLC Number: 

  • TP391
[1] BERTALMIO,SAPIRO G.Image inpainting [C]//Proceedings of International Conference on Computer Graphics an Interactive Techniques.New Orleans,Louisiana,USA,2000:417-424.
[2] CRIMINISI A,PEREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[3] HINTON G E.A Practical Guide to Training Restricted Boltzmann Machines[J].Momentum,2012,9(1):599-619.
[4] BENGIO Y.Learning Deep Architectures for AI[J].Founda-tions and Trends in Machine Learning,2009,2(1):1-127.
[5] LIU G,REDA F A,SHIH K,et al.Image inpainting for irregular holes using partial convolutions[C]//European Conference on Computer Vision (ECCV) 2018.Computer Vision,2018(15):89-105.
[6] PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2016:2536-2544.
[7] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//International Conference on Neural Information Processing Systems.2014.
[8] RUMMELHART D E,HINTON G E,WILLIAMS R J.Learning Internal Representations by Error Propagation[J].Readings in Cognitive Science,1986,323(2):318-362.
[9] HINTON G E.Reducing the Dimensionality of Data with Neural Networks[J].Science,2006,313(5786):504-507.
[10] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet Classification with Deep Convolutional Neural Networks[C]//NIPS,2012.
[11] RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,Cham,2015.
[12] YU J H,LIN Z,YANG J M,et al.Free-Form Image Inpainting with Gated Convolution[C]//ARXIV.2018.
[13] YANG C,LU X,LIN Z,et al.High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis[C]//CVPR.2017:6721-6729.
[14] HONG X,XIONG P F,JI R H,et al.Deep Fusion Network for Image Completion[J].arXiv 1904.08060,2019.
[15] NAZERI K,NG E,JOSEPH T,et al.EdgeConnect:Generative Image Inpainting with AdversarialEdgeLearning[J].arXiv1901.00212,2019.
[16] YEH R A,CHEN C,LIM T Y,et al.Image Inpainting withDeep Generative Models[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).New York:IEEE Press,2017:5485-5489.
[17] RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[C]//Proc of International Conference on Learning Representations.2016:1-16.
[18] IIZUKA S,SIMO-SERRA E,ISHIKAWA H .Globally and locally consistent image completion[J].ACM Transactions on Graphics,2017,36(4):1-14.
[19] DOLHANSKY B,FERRER C C,WAY H,et al.Eye In-Painting with Exemplar Generative Adversarial Networks[C]//CVPR.2018:00577.
[20] YU J H,LIN Z,YANG J M,et al.Generative Image Inpainting with Contextual Attention[C]//CVPR.2018.
[21] LI H F,LI G B,LIN L,et al.Context-Aware Semantic Inpainting[J].IEEE Transactions on Cybernetics (T-Cybernetics).DOI:10.1109/TCYB.2018.2865036,2019.
[22] ZHENG C X,CHAM T J,CAI J F.Pluralistic Image Completion[C]//CVPR.2019.
[23] CHEN K,QIAO Q,SONG Z J.Applications of Generative Adversarial Nets in Medical Image Processing[J].Life Science Instruments,2008,6:71-81.
[24] YANG G,YU S,DONG H,et al.DAGAN:Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction[J].IEEE Transactions on Medical Imaging,2018,37(6):1310-1321.
[25] SHITRIT O,RIKLIN RAVIV T.Accelerated Magnetic Reso-nance Imaging by Adversarial Neural Network[C]//Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support,2017:30-38.
[26] QUAN T M,NGUYEN-DUC T,JEONG W.Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss[J].IEEE Transactions on Medical Imaging,2018,37(6):1488-1497.
[27] BI L,KIM J,KUMAR A,et al.Synthesis of Positron Emission Tomography (PET) Images via Multi-channel Generative Adversarial Networks (GANs)[C]//Molecular Imaging,Recons-truction and Analysis of Moving Body Organs,and Stroke Imaging and Treatment.2017:43-51.
[28] WANG Y,YU B,WANG L,et al.3D conditional generative adversarial networks for high-quality PET image estimation at low dose[J].NeuroImage,2018,174:550-562.
[29] DENG Y,LOY C C,TANG X O.Aesthetic-driven Image En-hancement by Adversarial Learning[C]//2018 ACM Multimedia Conference.ACM,Amsterdam,Seoul,South Korea.New York:ACM,2018:870-878.
[30] PERRONNIN F.AVA:A Large-scale Database for AestheticVisual Analysis[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition.RI,USA.New Jersey:IEEE,2012:2408-2415.
[31] LU Q W,TAO Q C,ZHAO Y L,et al.Simplified Based onComic Draft Drawings Generated Against the Web[J].Acta Automatica Sinica,2018,44(5):840-854.
[32] DONG C,LOY C C,HE K M,et al.Image superresolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-307.
[33] LIANG Y,WANG J J,ZHOU S P,et al.Incorporating imagepriors with deep convolutional neural networks for image super-resolution[J].Neurocomputing,2016,194:340-347.
[34] GU S H,ZUO W M,XIE Q,et al.Convolutional sparse coding for image super-resolution[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV).Santiago,Chile:IEEE,2015:1823-1831.
[35] YANG J C,WRIGHT J,HUANG T,et al.Image super-resolution as sparse representation of raw image patches[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition.Anchorage,Alaska,USA:IEEE,2008:1-8.
[36] TELEA A.An image inpainting technique based on the fastmarching method [J].Journal of Graphics Tools,2004,9(1):23-34.
[37] PETSCHNIGG G,SZELISKI R,AGRAWALA M,et al.Digital photography with flash and no-flash image pairs [J].ACM Trans on Graphics,2004,23(3):664-672.
[38] LAI K,BO L F,REN X F,et al.A large-scale hierar chical multi-view RGB-D object dataset [C]// 2011 IEEE International Conference on Robotics and Automation (ICRA).2011:1817-1824.
[39] ZHANG Y D,FUNKHOUSER T.Deep Depth Completion of a Single RGB-D Image[J].Computer Vision and Pattern Recognition (CVPR 2018),2018,3:175-185.
[40] LI S M,LEI G Q,FAN R.Depth Map Super-Resolution Based on Deep Convolutional Neural Networks[J].Acta Optica Sinica,2017,37(12):1210002.
[41] YU S X,HU L M,ZHANG J,et al.Depth image super-resolution reconstruction with two-channel pyramid convolutional neural networks[J].Application Research of Computers,2019,2:96.
[42] LEI Y M.The role of video surveillance images in detecting and solving crimes[J].Engineering Technology,2016(12):302-302.
[43] WANG X D,WEI H Q,GAO C,et al.Identity preserving face complexion with generative adversarial networks[J].Chinese Journal of Netword and Information Security,2018,4(8):71-76.
[44] DING Y D,YU B.Application Research of New Computer Vision Technology in Old Film Restoration[J].Advanced Motion Picture Technology,2018,8:8.
[45] XIA T R,YU Y D,YU B.An old film restoration method using subframe quilting[J].Journal of Shanghai University,2018,24(4):503-511.
[46] SATOSHI I,EDGAR S S,ISHIKAWA H.Let there be Color!:Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification[J].Acm Transactions on Graphics,2016,35(4):1-11.
[47] ZHU Z.Image Enhancement and Analysis for Street Views[D].Tsinghua University,2016.
[1] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[2] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[3] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[4] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[5] HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[6] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[7] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[8] HU Yan-yu, ZHAO Long, DONG Xiang-jun. Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification [J]. Computer Science, 2022, 49(7): 73-78.
[9] CHENG Cheng, JIANG Ai-lian. Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction [J]. Computer Science, 2022, 49(7): 120-126.
[10] HOU Yu-tao, ABULIZI Abudukelimu, ABUDUKELIMU Halidanmu. Advances in Chinese Pre-training Models [J]. Computer Science, 2022, 49(7): 148-163.
[11] ZHOU Hui, SHI Hao-chen, TU Yao-feng, HUANG Sheng-jun. Robust Deep Neural Network Learning Based on Active Sampling [J]. Computer Science, 2022, 49(7): 164-169.
[12] SU Dan-ning, CAO Gui-tao, WANG Yan-nan, WANG Hong, REN He. Survey of Deep Learning for Radar Emitter Identification Based on Small Sample [J]. Computer Science, 2022, 49(7): 226-235.
[13] WANG Jun-feng, LIU Fan, YANG Sai, LYU Tan-yue, CHEN Zhi-yu, XU Feng. Dam Crack Detection Based on Multi-source Transfer Learning [J]. Computer Science, 2022, 49(6A): 319-324.
[14] CHU Yu-chun, GONG Hang, Wang Xue-fang, LIU Pei-shun. Study on Knowledge Distillation of Target Detection Algorithm Based on YOLOv4 [J]. Computer Science, 2022, 49(6A): 337-344.
[15] LIU Wei-ye, LU Hui-min, LI Yu-peng, MA Ning. Survey on Finger Vein Recognition Research [J]. Computer Science, 2022, 49(6A): 1-11.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!