Computer Science ›› 2021, Vol. 48 ›› Issue (3): 14-26.doi: 10.11896/jsjkx.210100048

Special Issue: Advances on Multimedia Technology

• Advances on Multimedia Technology • Previous Articles     Next Articles

Survey on Image Inpainting Research Progress

ZHAO Lu-lu1, SHEN Ling2, HONG Ri-chang1   

  1. 1 School of Computer Science and Information Engineering,Hefei University of Technology,Hefei 230601,China
    2 School of Internet,Anhui University,Hefei 230039,China
  • Received:2021-01-06 Revised:2021-01-26 Online:2021-03-15 Published:2021-03-05
  • About author:
    ZHAO Lu-lu,born in 1997,postgra-duate.Her main research interests include deep learning and image inpain-ting.
    SHEN Ling,born in 1983,Ph.D,lectu-rer.Her main research interests include computer vision and machine learning.
  • Supported by:
    Key Program of the National Natural Science Foundation of China(61932009).

Abstract: Image inpainting is a challenging research topic in the field of computer vision.In recent years,the development of deep learning technology has promoted the significant improvement in the performance of image inpainting,which makes image inpainting a traditional subject attracting extensive attention from scholars once again.This paper is dedicated to review the key technologies of image inpainting research.Due to the important role and far-reaching impact of deep learning technology in solving “large-area missing image inpainting”,this paper briefly introduces traditional image inpainting methods firstly,then focuses on inpainting models based on deep learning,mainly including model classification,comparison of advantages and disadvantages,scope of application and performance comparison on commonly used datasets,etc.Finally,the potential research directions and development trends of image inpainting are analyzed and prospected.

Key words: Auto encoder network, Convolutional neural network, Deep learning, Generative adversarial network, Image inpainting

CLC Number: 

  • TP391
[1]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[2]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[3]SZEGEDY C,LIU W,JIA Y,et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:1-9.
[4]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[5]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Gen-erative adversarial nets[C]//Advances in Neural Information Processing Systems.2014:2672-2680.
[6]QIANG Z P,HE L B,CHEN X,et al.Survey on deep learning image inpainting methods[J].Jurnal of Image and Graphics,2019,24(3):447-463.
[7]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.2000:417-424.
[8]CHAN T F,SHEN J.Nontexture inpainting by curvature-driven diffusions[J].Journal of Visual Communication and Image Representation,2001,12(4):436-449.
[9]SHEN J,CHAN T F.Mathematical models for local nontexture inpaintings[J].SIAM Journal on Applied Mathematics,2002,62(3):1019-1043.
[10]SHEN J,KANG S H,CHAN T F.Euler’s elastica and curvature-based inpainting[J].SIAM Journal on Applied Mathema-tics,2003,63(2):564-592.
[11]TSAI A,YEZZI A,WILLSKY A S.Curve evolution implementation of the Mumford-Shah functional for image segmentation,denoising,interpolation,and magnification[J].IEEE Transactions on Image Processing,2001,10(8):1169-1186.
[12]ESEDOGLU S.Digital Inpainting Based on The Mumford-Shah-Euler Image Model[J].European Journal of Applied Mathematics,2003,13(4):353-370.
[13]ZHANG H Y,PENG Q C.A survey on digital image inpainting [J].Journal of Image and Graphics,2007,12(1):1-10.
[14]GROSSAUER H.A combined PDE and texture synthesis ap-proach to inpainting[C]//European Conference on Computer Vision.Springer,Berlin,Heidelberg,2004:214-224.
[15]RANE S D,SAPIRO G,BERTALMIO M.Structure and texture filling-in of missing image blocks in wireless transmission and compression applications[J].IEEE Transactions on Image Processing,2003,12(3):296-303.
[16]YAMAUCHI H,HABER J,SEIDEL H P.Image restorationusing multiresolution texture synthesis and image inpainting[C]//Proceedings Computer Graphics International 2003.IEEE,2003:120-125.
[17]DRORI I,COHEN-OR D,YESHURUN H.Fragment-based image completion[J].ACM Transactions on Graphics,2003,22(3):303-312.
[18]ZHANG Y,XIAO J,SHAH M.Region completion in a single image[OL].http://diglib.eg.org/handle/10.2312/egs20041002.
[19]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[20]TANG F,YING Y,WANG J,et al.A novel texture synthesis based algorithm for object removal in photographs[C]//Annual Asian Computing Science Conference.Springer,Berlin,Heidelberg,2004:248-258.
[21]CRIMINISI A,PEREZ P,TOYAMA K.Object removal by exemplar-based inpainting[C]//2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2003,2:II-II.
[22]CHENG W H,HSIEH C W,LIN S K,et al.Robust algorithm for exemplar-based image inpainting[C]//Proceedings of International Conference on Computer Graphics,Imaging and Visua-lization.2005:64-69.
[23]HAYS J,EFROS A A.Scene completion using millions of photographs[J].Communications of the ACM ,2008,51(10):87-94.
[24]WHYTE O,SIVIC J,ZISSERMAN A.Get Out of my Picture! Internet-based Inpainting[C]//BMVC.2009,2(4):5.
[25]LIU Y,SHE J C,GONG Y C,et al.Survey of facial completion techniques based on deep learning[J].Application Research of Computers,2021,38(1):9-14.
[26]RUMELHART D E,HINTON G E,WILLIAMS R J.Learning internal representations by error propagation[R].California Univ San Diego La Jolla Inst for Cognitive Science,1985.
[27]HINTON G E,OSINDERO S,TEH Y W.A fast learning algorithm for deep belief nets[J].Neural computation,2006,18(7):1527-1554.
[28]MASCI J,MEIER U,CIREŞAN D,et al.Stacked convolutional auto-encoders for hierarchical feature extraction[C]//International Conference on Artificial Neural Networks.Springer,Berlin,Heidelberg,2011:52-59.
[29]BENGIO Y,LAMBLIN P,POPOVICI D,et al.Greedy layer-wise training of deep networks[J].Advances in Neural Information Processing Systems,2006,19:153-160.
[30]KINGMA D P,WELLING M.Auto-encoding variational bayes[J].arXiv:1312.6114,2013.
[31]SHEN L.Research on image inpainting methods based on semantic perception deep model [D].Hefei:Hefei University of Technology,2020.
[32]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[33]IIZUKA S,SIMO-SERRA E,ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics (ToG),2017,36(4):1-14.
[34]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irre-gular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:85-100.
[35]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Compu-ter-assisted Intervention.Springer,Cham,2015:234-241.
[36]YU J,LIN Z,YANG J,et al.Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE International Conference on Compu-ter Vision.2019:4471-4480.
[37]XIE C,LIU S,LI C,et al.Image inpainting with learnable bidirectional attention maps[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:8858-8867.
[38]WANG Y,TAO X,QI X,et al.Image inpainting via generative multi-column convolutional neural networks[C]//Advances in Neural Information Processing Systems.2018:331-340.
[39]XIAO Q,LI G,CHEN Q.Deep inception generative network for cognitive image inpainting[J].arXiv:1812.01458,2018.
[40]HUI Z,LI J,WANG X,et al.Image fine-grained inpainting[J].arXiv:2002.02609,2020.
[41]ZHANG H,HU Z,LUO C,et al.Semantic image inpaintingwith progressive generative networks[C]//Proceedings of the 26th ACM International Conference on Multimedia.2018:1939-1947.
[42]HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural Computation,1997,9(8):1735-1780.
[43]SHEN L,HONG R,ZHANG H,et al.Single-shot Semantic Image Inpainting with Densely Connected Generative Networks[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019:1861-1869.
[44]HONG X,XIONG P,JI R,et al.Deep fusion network for image completion[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019:2033-2042.
[45]YU T,GUO Z,JIN X,et al.Region Normalization for Image Inpainting[C]//AAAI.2020:12733-12740.
[46]NAZERI K,NG E,JOSEPH T,et al.Edgeconnect:Generative image inpainting with adversarial edge learning[J].arXiv:1901.00212,2019.
[47]CAI N,SU Z,LIN Z,et al.Blind inpainting using the fully con-volutional neural network[J].The Visual Computer,2017,33(2):249-261.
[48]LIU Y,PAN J,SU Z.Deep blind image inpainting[C]//International Conference on Intelligent Science and Big Data Enginee-ring.Springer,Cham,2019:128-141.
[49]WANG Y,CHEN Y C,TAO X,et al.VCNet:A Robust Approach to Blind Image Inpainting[J].arXiv:2003.06816,2020.
[50]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6721-6729.
[51]SONG Y,YANG C,LIN Z,et al.Contextual-based image in-painting:Infer,match,and translate[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:3-19.
[52]YAN Z,LI X,LI M,et al.Shift-net:Image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:1-17.
[53]YU J,LIN Z,YANG J,et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:5505-5514.
[54]LIU H,JIANG B,XIAO Y,et al.Coherent semantic attention for image inpainting[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:4170-4179.
[55]SAGONG M,SHIN Y,KIM S,et al.Pepsi:Fast image inpainting with parallel decoding network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:11360-11368.
[56]SHIN Y G,SAGONG M C,YEO Y J,et al.Pepsi++:Fast and lightweight network for image inpainting[J].IEEE Transactions on Neural Networks and Learning Systems,2020,32(1):252-265.
[57]ZENG Y,FU J,CHAO H,et al.Learning pyramid-context encoder network for high-quality image inpainting[C]//Procee-dings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:1486-1494.
[58]LIU H,JIANG B,SONG Y,et al.Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations[J].arXiv:2007.06929,2020.
[59]LI J,WANG N,ZHANG L,et al.Recurrent Feature Reasoning for Image Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7760-7768.
[60]ZENG Y,LIN Z,YANG J,et al.High-Resolution Image In-painting with Iterative Confidence Feedback and Guided Upsampling[J].arXiv:2005.11742,2020.
[61]YI Z,TANG Q,AZIZI S,et al.Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7508-7517.
[62]SONG Y,YANG C,SHEN Y,et al.Spg-net:Segmentation prediction and guidance network for image inpainting[J].arXiv:1805.03356,2018.
[63]XIONG W,YU J,LIN Z,et al.Foreground-aware image inpain-ting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:5840-5848.
[64]LIAO L,XIAO J,WANG Z,et al.Guidance and Evaluation:Semantic-Aware Image Inpainting for Mixed Scenes[J].arXiv:2003.06877,2020.
[65]LI J,HE F,ZHANG L,et al.Progressive reconstruction of vi-sual structure for image inpainting[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:5962-5971.
[66]REN Y,YU X,ZHANG R,et al.Structureflow:Image inpainting via structure-aware appearance flow[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:181-190.
[67]XU L,LU C,XU Y,et al.Image smoothing via L 0 gradient minimization[C]//Proceedings of the 2011 SIGGRAPH Asia Conference.2011:1-12.
[68]XU L,YAN Q,XIA Y,et al.Structure extraction from texture via relative total variation[J].ACM Transactions on Graphics (TOG),2012,31(6):1-10.
[69]ZHOU T,TULSIANI S,SUN W,et al.View synthesis by appearance flow[C]//European Conference on Computer Vision.Springer,Cham,2016:286-301.
[70]YANG J,QI Z,SHI Y.Learning to Incorporate StructureKnowledge for Image Inpainting[C]//AAAI.2020:12605-12612.
[71]OORD A,KALCHBRENNER N,KAVUKCUOGLU K.Pixel recurrent neural networks[J].arXiv:1601.06759,2016.
[72]YEH R A,CHEN C,YIAN LIM T,et al.Semantic image inpainting with deep generative models[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:5485-5493.
[73]BAO J,CHEN D,WEN F,et al.CVAE-GAN:fine-grained image generation through asymmetric training[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:2745-2754.
[74]LAHIRI A,JAIN A K,AGRAWAL S,et al.Prior Guided GAN Based Semantic Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:13696-13705.
[75]ZHENG C,CHAM T J,CAI J.Pluralistic image completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:1438-1447.
[76]ZHAO L,MO Q,LIN S,et al.UCTGAN:Diverse Image In-painting Based on Unsupervised Cross-Space Translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:5741-5750.
[77]ULYANOV D,VEDALDI A,LEMPITSKY V.Deep imageprior[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:9446-9454.
[78]RUSSAKOVSKY O,DENG J,SU H,et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115(3):211-252.
[79]DENG J,DONG W,SOCHER R,et al.Imagenet:A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2009:248-255.
[80]DOERSCH C,SINGH S,GUPTA A,et al.What Makes Paris Look Like Paris?[J].Acm Transactions on Graphics,2012,31(4):1-9.
[81]ZHOU B,LAPEDRIZA A,KHOSLA A,et al.Places:A 10 million image database for scene recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,40(6):1452-1464.
[82]LIU Z,LUO P,WANG X,et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision.2015:3730-3738.
[83]KARRAS T,AILA T,LAINE S,et al.Progressive growing of gans for improved quality,stability,and variation[J].arXiv:1710.10196,2017.
[1] ZHANG Jia, DONG Shou-bin. Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer [J]. Computer Science, 2022, 49(9): 41-47.
[2] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[3] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[4] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[5] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[6] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[7] HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[8] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[9] CHEN Yong-quan, JIANG Ying. Analysis Method of APP User Behavior Based on Convolutional Neural Network [J]. Computer Science, 2022, 49(8): 78-85.
[10] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[11] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[12] HOU Yu-tao, ABULIZI Abudukelimu, ABUDUKELIMU Halidanmu. Advances in Chinese Pre-training Models [J]. Computer Science, 2022, 49(7): 148-163.
[13] ZHOU Hui, SHI Hao-chen, TU Yao-feng, HUANG Sheng-jun. Robust Deep Neural Network Learning Based on Active Sampling [J]. Computer Science, 2022, 49(7): 164-169.
[14] SU Dan-ning, CAO Gui-tao, WANG Yan-nan, WANG Hong, REN He. Survey of Deep Learning for Radar Emitter Identification Based on Small Sample [J]. Computer Science, 2022, 49(7): 226-235.
[15] HU Yan-yu, ZHAO Long, DONG Xiang-jun. Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification [J]. Computer Science, 2022, 49(7): 73-78.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!