计算机科学 ›› 2021, Vol. 48 ›› Issue (3): 14-26.doi: 10.11896/jsjkx.210100048

所属专题: 多媒体技术进展

• 多媒体技术进展* 上一篇    下一篇

图像修复研究进展综述

赵露露1, 沈玲2, 洪日昌1   

  1. 1 合肥工业大学计算机与信息学院 合肥230601
    2 安徽大学互联网学院 合肥230039
  • 收稿日期:2021-01-06 修回日期:2021-01-26 出版日期:2021-03-15 发布日期:2021-03-05
  • 通讯作者: 沈玲(sammiling315@gmail.com)
  • 作者简介:luluzhao@mail.hfut.edu.cn
  • 基金资助:
    国家自然科学基金重点项目(61932009)

Survey on Image Inpainting Research Progress

ZHAO Lu-lu1, SHEN Ling2, HONG Ri-chang1   

  1. 1 School of Computer Science and Information Engineering,Hefei University of Technology,Hefei 230601,China
    2 School of Internet,Anhui University,Hefei 230039,China
  • Received:2021-01-06 Revised:2021-01-26 Online:2021-03-15 Published:2021-03-05
  • About author:
    ZHAO Lu-lu,born in 1997,postgra-duate.Her main research interests include deep learning and image inpain-ting.
    SHEN Ling,born in 1983,Ph.D,lectu-rer.Her main research interests include computer vision and machine learning.
  • Supported by:
    Key Program of the National Natural Science Foundation of China(61932009).

摘要: 图像修复是计算机视觉领域中极具挑战性的研究课题。近年来,深度学习技术的发展推动了图像修复性能的显著提升,使得图像修复这一传统课题再次引起了学者们的广泛关注。文章致力于综述图像修复研究的关键技术。由于深度学习技术在解决“大面积缺失图像修复”问题时具有重要作用并带来了深远影响,文中在简要介绍传统图像修复方法的基础上,重点介绍了基于深度学习的修复模型,主要包括模型分类、优缺点对比、适用范围和在常用数据集上的性能对比等,最后对图像修复潜在的研究方向和发展动态进行了分析和展望。

关键词: 卷积神经网络, 深度学习, 生成对抗网络, 图像修复, 自编码网络

Abstract: Image inpainting is a challenging research topic in the field of computer vision.In recent years,the development of deep learning technology has promoted the significant improvement in the performance of image inpainting,which makes image inpainting a traditional subject attracting extensive attention from scholars once again.This paper is dedicated to review the key technologies of image inpainting research.Due to the important role and far-reaching impact of deep learning technology in solving “large-area missing image inpainting”,this paper briefly introduces traditional image inpainting methods firstly,then focuses on inpainting models based on deep learning,mainly including model classification,comparison of advantages and disadvantages,scope of application and performance comparison on commonly used datasets,etc.Finally,the potential research directions and development trends of image inpainting are analyzed and prospected.

Key words: Auto encoder network, Convolutional neural network, Deep learning, Generative adversarial network, Image inpainting

中图分类号: 

  • TP391
[1]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[2]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[3]SZEGEDY C,LIU W,JIA Y,et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:1-9.
[4]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[5]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Gen-erative adversarial nets[C]//Advances in Neural Information Processing Systems.2014:2672-2680.
[6]QIANG Z P,HE L B,CHEN X,et al.Survey on deep learning image inpainting methods[J].Jurnal of Image and Graphics,2019,24(3):447-463.
[7]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.2000:417-424.
[8]CHAN T F,SHEN J.Nontexture inpainting by curvature-driven diffusions[J].Journal of Visual Communication and Image Representation,2001,12(4):436-449.
[9]SHEN J,CHAN T F.Mathematical models for local nontexture inpaintings[J].SIAM Journal on Applied Mathematics,2002,62(3):1019-1043.
[10]SHEN J,KANG S H,CHAN T F.Euler’s elastica and curvature-based inpainting[J].SIAM Journal on Applied Mathema-tics,2003,63(2):564-592.
[11]TSAI A,YEZZI A,WILLSKY A S.Curve evolution implementation of the Mumford-Shah functional for image segmentation,denoising,interpolation,and magnification[J].IEEE Transactions on Image Processing,2001,10(8):1169-1186.
[12]ESEDOGLU S.Digital Inpainting Based on The Mumford-Shah-Euler Image Model[J].European Journal of Applied Mathematics,2003,13(4):353-370.
[13]ZHANG H Y,PENG Q C.A survey on digital image inpainting [J].Journal of Image and Graphics,2007,12(1):1-10.
[14]GROSSAUER H.A combined PDE and texture synthesis ap-proach to inpainting[C]//European Conference on Computer Vision.Springer,Berlin,Heidelberg,2004:214-224.
[15]RANE S D,SAPIRO G,BERTALMIO M.Structure and texture filling-in of missing image blocks in wireless transmission and compression applications[J].IEEE Transactions on Image Processing,2003,12(3):296-303.
[16]YAMAUCHI H,HABER J,SEIDEL H P.Image restorationusing multiresolution texture synthesis and image inpainting[C]//Proceedings Computer Graphics International 2003.IEEE,2003:120-125.
[17]DRORI I,COHEN-OR D,YESHURUN H.Fragment-based image completion[J].ACM Transactions on Graphics,2003,22(3):303-312.
[18]ZHANG Y,XIAO J,SHAH M.Region completion in a single image[OL].http://diglib.eg.org/handle/10.2312/egs20041002.
[19]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[20]TANG F,YING Y,WANG J,et al.A novel texture synthesis based algorithm for object removal in photographs[C]//Annual Asian Computing Science Conference.Springer,Berlin,Heidelberg,2004:248-258.
[21]CRIMINISI A,PEREZ P,TOYAMA K.Object removal by exemplar-based inpainting[C]//2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2003,2:II-II.
[22]CHENG W H,HSIEH C W,LIN S K,et al.Robust algorithm for exemplar-based image inpainting[C]//Proceedings of International Conference on Computer Graphics,Imaging and Visua-lization.2005:64-69.
[23]HAYS J,EFROS A A.Scene completion using millions of photographs[J].Communications of the ACM ,2008,51(10):87-94.
[24]WHYTE O,SIVIC J,ZISSERMAN A.Get Out of my Picture! Internet-based Inpainting[C]//BMVC.2009,2(4):5.
[25]LIU Y,SHE J C,GONG Y C,et al.Survey of facial completion techniques based on deep learning[J].Application Research of Computers,2021,38(1):9-14.
[26]RUMELHART D E,HINTON G E,WILLIAMS R J.Learning internal representations by error propagation[R].California Univ San Diego La Jolla Inst for Cognitive Science,1985.
[27]HINTON G E,OSINDERO S,TEH Y W.A fast learning algorithm for deep belief nets[J].Neural computation,2006,18(7):1527-1554.
[28]MASCI J,MEIER U,CIREŞAN D,et al.Stacked convolutional auto-encoders for hierarchical feature extraction[C]//International Conference on Artificial Neural Networks.Springer,Berlin,Heidelberg,2011:52-59.
[29]BENGIO Y,LAMBLIN P,POPOVICI D,et al.Greedy layer-wise training of deep networks[J].Advances in Neural Information Processing Systems,2006,19:153-160.
[30]KINGMA D P,WELLING M.Auto-encoding variational bayes[J].arXiv:1312.6114,2013.
[31]SHEN L.Research on image inpainting methods based on semantic perception deep model [D].Hefei:Hefei University of Technology,2020.
[32]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[33]IIZUKA S,SIMO-SERRA E,ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics (ToG),2017,36(4):1-14.
[34]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irre-gular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:85-100.
[35]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Compu-ter-assisted Intervention.Springer,Cham,2015:234-241.
[36]YU J,LIN Z,YANG J,et al.Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE International Conference on Compu-ter Vision.2019:4471-4480.
[37]XIE C,LIU S,LI C,et al.Image inpainting with learnable bidirectional attention maps[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:8858-8867.
[38]WANG Y,TAO X,QI X,et al.Image inpainting via generative multi-column convolutional neural networks[C]//Advances in Neural Information Processing Systems.2018:331-340.
[39]XIAO Q,LI G,CHEN Q.Deep inception generative network for cognitive image inpainting[J].arXiv:1812.01458,2018.
[40]HUI Z,LI J,WANG X,et al.Image fine-grained inpainting[J].arXiv:2002.02609,2020.
[41]ZHANG H,HU Z,LUO C,et al.Semantic image inpaintingwith progressive generative networks[C]//Proceedings of the 26th ACM International Conference on Multimedia.2018:1939-1947.
[42]HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural Computation,1997,9(8):1735-1780.
[43]SHEN L,HONG R,ZHANG H,et al.Single-shot Semantic Image Inpainting with Densely Connected Generative Networks[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019:1861-1869.
[44]HONG X,XIONG P,JI R,et al.Deep fusion network for image completion[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019:2033-2042.
[45]YU T,GUO Z,JIN X,et al.Region Normalization for Image Inpainting[C]//AAAI.2020:12733-12740.
[46]NAZERI K,NG E,JOSEPH T,et al.Edgeconnect:Generative image inpainting with adversarial edge learning[J].arXiv:1901.00212,2019.
[47]CAI N,SU Z,LIN Z,et al.Blind inpainting using the fully con-volutional neural network[J].The Visual Computer,2017,33(2):249-261.
[48]LIU Y,PAN J,SU Z.Deep blind image inpainting[C]//International Conference on Intelligent Science and Big Data Enginee-ring.Springer,Cham,2019:128-141.
[49]WANG Y,CHEN Y C,TAO X,et al.VCNet:A Robust Approach to Blind Image Inpainting[J].arXiv:2003.06816,2020.
[50]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6721-6729.
[51]SONG Y,YANG C,LIN Z,et al.Contextual-based image in-painting:Infer,match,and translate[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:3-19.
[52]YAN Z,LI X,LI M,et al.Shift-net:Image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:1-17.
[53]YU J,LIN Z,YANG J,et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:5505-5514.
[54]LIU H,JIANG B,XIAO Y,et al.Coherent semantic attention for image inpainting[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:4170-4179.
[55]SAGONG M,SHIN Y,KIM S,et al.Pepsi:Fast image inpainting with parallel decoding network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:11360-11368.
[56]SHIN Y G,SAGONG M C,YEO Y J,et al.Pepsi++:Fast and lightweight network for image inpainting[J].IEEE Transactions on Neural Networks and Learning Systems,2020,32(1):252-265.
[57]ZENG Y,FU J,CHAO H,et al.Learning pyramid-context encoder network for high-quality image inpainting[C]//Procee-dings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:1486-1494.
[58]LIU H,JIANG B,SONG Y,et al.Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations[J].arXiv:2007.06929,2020.
[59]LI J,WANG N,ZHANG L,et al.Recurrent Feature Reasoning for Image Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7760-7768.
[60]ZENG Y,LIN Z,YANG J,et al.High-Resolution Image In-painting with Iterative Confidence Feedback and Guided Upsampling[J].arXiv:2005.11742,2020.
[61]YI Z,TANG Q,AZIZI S,et al.Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7508-7517.
[62]SONG Y,YANG C,SHEN Y,et al.Spg-net:Segmentation prediction and guidance network for image inpainting[J].arXiv:1805.03356,2018.
[63]XIONG W,YU J,LIN Z,et al.Foreground-aware image inpain-ting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:5840-5848.
[64]LIAO L,XIAO J,WANG Z,et al.Guidance and Evaluation:Semantic-Aware Image Inpainting for Mixed Scenes[J].arXiv:2003.06877,2020.
[65]LI J,HE F,ZHANG L,et al.Progressive reconstruction of vi-sual structure for image inpainting[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:5962-5971.
[66]REN Y,YU X,ZHANG R,et al.Structureflow:Image inpainting via structure-aware appearance flow[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:181-190.
[67]XU L,LU C,XU Y,et al.Image smoothing via L 0 gradient minimization[C]//Proceedings of the 2011 SIGGRAPH Asia Conference.2011:1-12.
[68]XU L,YAN Q,XIA Y,et al.Structure extraction from texture via relative total variation[J].ACM Transactions on Graphics (TOG),2012,31(6):1-10.
[69]ZHOU T,TULSIANI S,SUN W,et al.View synthesis by appearance flow[C]//European Conference on Computer Vision.Springer,Cham,2016:286-301.
[70]YANG J,QI Z,SHI Y.Learning to Incorporate StructureKnowledge for Image Inpainting[C]//AAAI.2020:12605-12612.
[71]OORD A,KALCHBRENNER N,KAVUKCUOGLU K.Pixel recurrent neural networks[J].arXiv:1601.06759,2016.
[72]YEH R A,CHEN C,YIAN LIM T,et al.Semantic image inpainting with deep generative models[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:5485-5493.
[73]BAO J,CHEN D,WEN F,et al.CVAE-GAN:fine-grained image generation through asymmetric training[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:2745-2754.
[74]LAHIRI A,JAIN A K,AGRAWAL S,et al.Prior Guided GAN Based Semantic Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:13696-13705.
[75]ZHENG C,CHAM T J,CAI J.Pluralistic image completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:1438-1447.
[76]ZHAO L,MO Q,LIN S,et al.UCTGAN:Diverse Image In-painting Based on Unsupervised Cross-Space Translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:5741-5750.
[77]ULYANOV D,VEDALDI A,LEMPITSKY V.Deep imageprior[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:9446-9454.
[78]RUSSAKOVSKY O,DENG J,SU H,et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115(3):211-252.
[79]DENG J,DONG W,SOCHER R,et al.Imagenet:A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2009:248-255.
[80]DOERSCH C,SINGH S,GUPTA A,et al.What Makes Paris Look Like Paris?[J].Acm Transactions on Graphics,2012,31(4):1-9.
[81]ZHOU B,LAPEDRIZA A,KHOSLA A,et al.Places:A 10 million image database for scene recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,40(6):1452-1464.
[82]LIU Z,LUO P,WANG X,et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision.2015:3730-3738.
[83]KARRAS T,AILA T,LAINE S,et al.Progressive growing of gans for improved quality,stability,and variation[J].arXiv:1710.10196,2017.
[1] 饶志双, 贾真, 张凡, 李天瑞.
基于Key-Value关联记忆网络的知识图谱问答方法
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[2] 汤凌韬, 王迪, 张鲁飞, 刘盛云.
基于安全多方计算和差分隐私的联邦学习方案
Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy
计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[3] 张佳, 董守斌.
基于评论方面级用户偏好迁移的跨领域推荐算法
Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer
计算机科学, 2022, 49(9): 41-47. https://doi.org/10.11896/jsjkx.220200131
[4] 周乐员, 张剑华, 袁甜甜, 陈胜勇.
多层注意力机制融合的序列到序列中国连续手语识别和翻译
Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion
计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[5] 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺.
时序知识图谱表示学习
Temporal Knowledge Graph Representation Learning
计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[6] 李宗民, 张玉鹏, 刘玉杰, 李华.
基于可变形图卷积的点云表征学习
Deformable Graph Convolutional Networks Based Point Cloud Representation Learning
计算机科学, 2022, 49(8): 273-278. https://doi.org/10.11896/jsjkx.210900023
[7] 王剑, 彭雨琦, 赵宇斐, 杨健.
基于深度学习的社交网络舆情信息抽取方法综述
Survey of Social Network Public Opinion Information Extraction Based on Deep Learning
计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[8] 郝志荣, 陈龙, 黄嘉成.
面向文本分类的类别区分式通用对抗攻击方法
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[9] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
基于改进位置编码的谣言检测模型
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[10] 陈泳全, 姜瑛.
基于卷积神经网络的APP用户行为分析方法
Analysis Method of APP User Behavior Based on Convolutional Neural Network
计算机科学, 2022, 49(8): 78-85. https://doi.org/10.11896/jsjkx.210700121
[11] 朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥.
基于注意力机制的医学影像深度哈希检索算法
Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism
计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[12] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[13] 檀莹莹, 王俊丽, 张超波.
基于图卷积神经网络的文本分类方法研究综述
Review of Text Classification Methods Based on Graph Convolutional Network
计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064
[14] 胡艳羽, 赵龙, 董祥军.
一种用于癌症分类的两阶段深度特征选择提取算法
Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification
计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[15] 张颖涛, 张杰, 张睿, 张文强.
全局信息引导的真实图像风格迁移
Photorealistic Style Transfer Guided by Global Information
计算机科学, 2022, 49(7): 100-105. https://doi.org/10.11896/jsjkx.210600036
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!