计算机科学 ›› 2021, Vol. 48 ›› Issue (3): 14-26.doi: 10.11896/jsjkx.210100048
所属专题: 多媒体技术进展
赵露露1, 沈玲2, 洪日昌1
ZHAO Lu-lu1, SHEN Ling2, HONG Ri-chang1
摘要: 图像修复是计算机视觉领域中极具挑战性的研究课题。近年来,深度学习技术的发展推动了图像修复性能的显著提升,使得图像修复这一传统课题再次引起了学者们的广泛关注。文章致力于综述图像修复研究的关键技术。由于深度学习技术在解决“大面积缺失图像修复”问题时具有重要作用并带来了深远影响,文中在简要介绍传统图像修复方法的基础上,重点介绍了基于深度学习的修复模型,主要包括模型分类、优缺点对比、适用范围和在常用数据集上的性能对比等,最后对图像修复潜在的研究方向和发展动态进行了分析和展望。
中图分类号:
[1]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90. [2]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014. [3]SZEGEDY C,LIU W,JIA Y,et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:1-9. [4]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778. [5]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Gen-erative adversarial nets[C]//Advances in Neural Information Processing Systems.2014:2672-2680. [6]QIANG Z P,HE L B,CHEN X,et al.Survey on deep learning image inpainting methods[J].Jurnal of Image and Graphics,2019,24(3):447-463. [7]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.2000:417-424. [8]CHAN T F,SHEN J.Nontexture inpainting by curvature-driven diffusions[J].Journal of Visual Communication and Image Representation,2001,12(4):436-449. [9]SHEN J,CHAN T F.Mathematical models for local nontexture inpaintings[J].SIAM Journal on Applied Mathematics,2002,62(3):1019-1043. [10]SHEN J,KANG S H,CHAN T F.Euler’s elastica and curvature-based inpainting[J].SIAM Journal on Applied Mathema-tics,2003,63(2):564-592. [11]TSAI A,YEZZI A,WILLSKY A S.Curve evolution implementation of the Mumford-Shah functional for image segmentation,denoising,interpolation,and magnification[J].IEEE Transactions on Image Processing,2001,10(8):1169-1186. [12]ESEDOGLU S.Digital Inpainting Based on The Mumford-Shah-Euler Image Model[J].European Journal of Applied Mathematics,2003,13(4):353-370. [13]ZHANG H Y,PENG Q C.A survey on digital image inpainting [J].Journal of Image and Graphics,2007,12(1):1-10. [14]GROSSAUER H.A combined PDE and texture synthesis ap-proach to inpainting[C]//European Conference on Computer Vision.Springer,Berlin,Heidelberg,2004:214-224. [15]RANE S D,SAPIRO G,BERTALMIO M.Structure and texture filling-in of missing image blocks in wireless transmission and compression applications[J].IEEE Transactions on Image Processing,2003,12(3):296-303. [16]YAMAUCHI H,HABER J,SEIDEL H P.Image restorationusing multiresolution texture synthesis and image inpainting[C]//Proceedings Computer Graphics International 2003.IEEE,2003:120-125. [17]DRORI I,COHEN-OR D,YESHURUN H.Fragment-based image completion[J].ACM Transactions on Graphics,2003,22(3):303-312. [18]ZHANG Y,XIAO J,SHAH M.Region completion in a single image[OL].http://diglib.eg.org/handle/10.2312/egs20041002. [19]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212. [20]TANG F,YING Y,WANG J,et al.A novel texture synthesis based algorithm for object removal in photographs[C]//Annual Asian Computing Science Conference.Springer,Berlin,Heidelberg,2004:248-258. [21]CRIMINISI A,PEREZ P,TOYAMA K.Object removal by exemplar-based inpainting[C]//2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2003,2:II-II. [22]CHENG W H,HSIEH C W,LIN S K,et al.Robust algorithm for exemplar-based image inpainting[C]//Proceedings of International Conference on Computer Graphics,Imaging and Visua-lization.2005:64-69. [23]HAYS J,EFROS A A.Scene completion using millions of photographs[J].Communications of the ACM ,2008,51(10):87-94. [24]WHYTE O,SIVIC J,ZISSERMAN A.Get Out of my Picture! Internet-based Inpainting[C]//BMVC.2009,2(4):5. [25]LIU Y,SHE J C,GONG Y C,et al.Survey of facial completion techniques based on deep learning[J].Application Research of Computers,2021,38(1):9-14. [26]RUMELHART D E,HINTON G E,WILLIAMS R J.Learning internal representations by error propagation[R].California Univ San Diego La Jolla Inst for Cognitive Science,1985. [27]HINTON G E,OSINDERO S,TEH Y W.A fast learning algorithm for deep belief nets[J].Neural computation,2006,18(7):1527-1554. [28]MASCI J,MEIER U,CIREŞAN D,et al.Stacked convolutional auto-encoders for hierarchical feature extraction[C]//International Conference on Artificial Neural Networks.Springer,Berlin,Heidelberg,2011:52-59. [29]BENGIO Y,LAMBLIN P,POPOVICI D,et al.Greedy layer-wise training of deep networks[J].Advances in Neural Information Processing Systems,2006,19:153-160. [30]KINGMA D P,WELLING M.Auto-encoding variational bayes[J].arXiv:1312.6114,2013. [31]SHEN L.Research on image inpainting methods based on semantic perception deep model [D].Hefei:Hefei University of Technology,2020. [32]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544. [33]IIZUKA S,SIMO-SERRA E,ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics (ToG),2017,36(4):1-14. [34]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irre-gular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:85-100. [35]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Compu-ter-assisted Intervention.Springer,Cham,2015:234-241. [36]YU J,LIN Z,YANG J,et al.Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE International Conference on Compu-ter Vision.2019:4471-4480. [37]XIE C,LIU S,LI C,et al.Image inpainting with learnable bidirectional attention maps[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:8858-8867. [38]WANG Y,TAO X,QI X,et al.Image inpainting via generative multi-column convolutional neural networks[C]//Advances in Neural Information Processing Systems.2018:331-340. [39]XIAO Q,LI G,CHEN Q.Deep inception generative network for cognitive image inpainting[J].arXiv:1812.01458,2018. [40]HUI Z,LI J,WANG X,et al.Image fine-grained inpainting[J].arXiv:2002.02609,2020. [41]ZHANG H,HU Z,LUO C,et al.Semantic image inpaintingwith progressive generative networks[C]//Proceedings of the 26th ACM International Conference on Multimedia.2018:1939-1947. [42]HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural Computation,1997,9(8):1735-1780. [43]SHEN L,HONG R,ZHANG H,et al.Single-shot Semantic Image Inpainting with Densely Connected Generative Networks[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019:1861-1869. [44]HONG X,XIONG P,JI R,et al.Deep fusion network for image completion[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019:2033-2042. [45]YU T,GUO Z,JIN X,et al.Region Normalization for Image Inpainting[C]//AAAI.2020:12733-12740. [46]NAZERI K,NG E,JOSEPH T,et al.Edgeconnect:Generative image inpainting with adversarial edge learning[J].arXiv:1901.00212,2019. [47]CAI N,SU Z,LIN Z,et al.Blind inpainting using the fully con-volutional neural network[J].The Visual Computer,2017,33(2):249-261. [48]LIU Y,PAN J,SU Z.Deep blind image inpainting[C]//International Conference on Intelligent Science and Big Data Enginee-ring.Springer,Cham,2019:128-141. [49]WANG Y,CHEN Y C,TAO X,et al.VCNet:A Robust Approach to Blind Image Inpainting[J].arXiv:2003.06816,2020. [50]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6721-6729. [51]SONG Y,YANG C,LIN Z,et al.Contextual-based image in-painting:Infer,match,and translate[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:3-19. [52]YAN Z,LI X,LI M,et al.Shift-net:Image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:1-17. [53]YU J,LIN Z,YANG J,et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:5505-5514. [54]LIU H,JIANG B,XIAO Y,et al.Coherent semantic attention for image inpainting[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:4170-4179. [55]SAGONG M,SHIN Y,KIM S,et al.Pepsi:Fast image inpainting with parallel decoding network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:11360-11368. [56]SHIN Y G,SAGONG M C,YEO Y J,et al.Pepsi++:Fast and lightweight network for image inpainting[J].IEEE Transactions on Neural Networks and Learning Systems,2020,32(1):252-265. [57]ZENG Y,FU J,CHAO H,et al.Learning pyramid-context encoder network for high-quality image inpainting[C]//Procee-dings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:1486-1494. [58]LIU H,JIANG B,SONG Y,et al.Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations[J].arXiv:2007.06929,2020. [59]LI J,WANG N,ZHANG L,et al.Recurrent Feature Reasoning for Image Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7760-7768. [60]ZENG Y,LIN Z,YANG J,et al.High-Resolution Image In-painting with Iterative Confidence Feedback and Guided Upsampling[J].arXiv:2005.11742,2020. [61]YI Z,TANG Q,AZIZI S,et al.Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7508-7517. [62]SONG Y,YANG C,SHEN Y,et al.Spg-net:Segmentation prediction and guidance network for image inpainting[J].arXiv:1805.03356,2018. [63]XIONG W,YU J,LIN Z,et al.Foreground-aware image inpain-ting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:5840-5848. [64]LIAO L,XIAO J,WANG Z,et al.Guidance and Evaluation:Semantic-Aware Image Inpainting for Mixed Scenes[J].arXiv:2003.06877,2020. [65]LI J,HE F,ZHANG L,et al.Progressive reconstruction of vi-sual structure for image inpainting[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:5962-5971. [66]REN Y,YU X,ZHANG R,et al.Structureflow:Image inpainting via structure-aware appearance flow[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:181-190. [67]XU L,LU C,XU Y,et al.Image smoothing via L 0 gradient minimization[C]//Proceedings of the 2011 SIGGRAPH Asia Conference.2011:1-12. [68]XU L,YAN Q,XIA Y,et al.Structure extraction from texture via relative total variation[J].ACM Transactions on Graphics (TOG),2012,31(6):1-10. [69]ZHOU T,TULSIANI S,SUN W,et al.View synthesis by appearance flow[C]//European Conference on Computer Vision.Springer,Cham,2016:286-301. [70]YANG J,QI Z,SHI Y.Learning to Incorporate StructureKnowledge for Image Inpainting[C]//AAAI.2020:12605-12612. [71]OORD A,KALCHBRENNER N,KAVUKCUOGLU K.Pixel recurrent neural networks[J].arXiv:1601.06759,2016. [72]YEH R A,CHEN C,YIAN LIM T,et al.Semantic image inpainting with deep generative models[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:5485-5493. [73]BAO J,CHEN D,WEN F,et al.CVAE-GAN:fine-grained image generation through asymmetric training[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:2745-2754. [74]LAHIRI A,JAIN A K,AGRAWAL S,et al.Prior Guided GAN Based Semantic Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:13696-13705. [75]ZHENG C,CHAM T J,CAI J.Pluralistic image completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:1438-1447. [76]ZHAO L,MO Q,LIN S,et al.UCTGAN:Diverse Image In-painting Based on Unsupervised Cross-Space Translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:5741-5750. [77]ULYANOV D,VEDALDI A,LEMPITSKY V.Deep imageprior[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:9446-9454. [78]RUSSAKOVSKY O,DENG J,SU H,et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115(3):211-252. [79]DENG J,DONG W,SOCHER R,et al.Imagenet:A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2009:248-255. [80]DOERSCH C,SINGH S,GUPTA A,et al.What Makes Paris Look Like Paris?[J].Acm Transactions on Graphics,2012,31(4):1-9. [81]ZHOU B,LAPEDRIZA A,KHOSLA A,et al.Places:A 10 million image database for scene recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,40(6):1452-1464. [82]LIU Z,LUO P,WANG X,et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision.2015:3730-3738. [83]KARRAS T,AILA T,LAINE S,et al.Progressive growing of gans for improved quality,stability,and variation[J].arXiv:1710.10196,2017. |
[1] | 饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277 |
[2] | 汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108 |
[3] | 张佳, 董守斌. 基于评论方面级用户偏好迁移的跨领域推荐算法 Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer 计算机科学, 2022, 49(9): 41-47. https://doi.org/10.11896/jsjkx.220200131 |
[4] | 周乐员, 张剑华, 袁甜甜, 陈胜勇. 多层注意力机制融合的序列到序列中国连续手语识别和翻译 Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion 计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026 |
[5] | 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204 |
[6] | 李宗民, 张玉鹏, 刘玉杰, 李华. 基于可变形图卷积的点云表征学习 Deformable Graph Convolutional Networks Based Point Cloud Representation Learning 计算机科学, 2022, 49(8): 273-278. https://doi.org/10.11896/jsjkx.210900023 |
[7] | 王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099 |
[8] | 郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077 |
[9] | 姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046 |
[10] | 陈泳全, 姜瑛. 基于卷积神经网络的APP用户行为分析方法 Analysis Method of APP User Behavior Based on Convolutional Neural Network 计算机科学, 2022, 49(8): 78-85. https://doi.org/10.11896/jsjkx.210700121 |
[11] | 朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥. 基于注意力机制的医学影像深度哈希检索算法 Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism 计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153 |
[12] | 孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061 |
[13] | 檀莹莹, 王俊丽, 张超波. 基于图卷积神经网络的文本分类方法研究综述 Review of Text Classification Methods Based on Graph Convolutional Network 计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064 |
[14] | 胡艳羽, 赵龙, 董祥军. 一种用于癌症分类的两阶段深度特征选择提取算法 Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification 计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092 |
[15] | 张颖涛, 张杰, 张睿, 张文强. 全局信息引导的真实图像风格迁移 Photorealistic Style Transfer Guided by Global Information 计算机科学, 2022, 49(7): 100-105. https://doi.org/10.11896/jsjkx.210600036 |
|