Computer Science ›› 2022, Vol. 49 ›› Issue (6): 217-223.doi: 10.11896/jsjkx.210500105

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Digital Mural Inpainting Method Based on Feature Perception

XU Hui1,2, KANG Jin-meng1, ZHANG Jia-wan1   

  1. 1 School of Computer Software,Tianjin University,Tianjin 300072,China
    2 Digitization Technology Research Center for Cultural Heritage Conservation and Promotion,Henan Institute of Science and Technology,Xinxiang,Henan 453000,China
  • Received:2021-05-14 Revised:2021-07-20 Online:2022-06-15 Published:2022-06-08
  • About author:XU Hui,born in 1986,Ph.D,is a member of China Computer Federation.Her main research interests include image synthesis and digital conservation of cultural heritage.
    ZHANG Jia-wan,born in 1975,professor,Ph.D supervisor,is a member of China Computer Federation.His main research interests include image synthesis,visualization and visual analysis.
  • Supported by:
    National Key R & D Program of China(2019YFC1521200).

Abstract: There are irregular damaged areas caused by various diseases of grottoes in Dunhuang murals,digital restoration is used to restore the image of the Dunhuang grotto murals,which will not cause damage to the original murals,but also get a better repair effect.Because of the large missing area in the mural mending,it cannot be realized by local non-semantic repair methods.Aiming at the restoration of the defective area of Dunhuang grotto murals,this paper designs an image repair method based on the generation of confrontation network,and uses semantically reasonable content to render the pixels in the missing area to realize the reconstruction of non-contact mural scenes,improve the efficiency of mural virtual restoration and the accuracy of restoration.The algorithm introduces a perceptual-loss function on the basis of generating an adversarial neural network,adds a three-layer convolutional layer to the generation model to collect image features of damaged areas,uses the perceptual loss to improve the model’s ability to repair high-frequency texture details,and uses extended convolution to extract range features,so as to stimulate the generative model to generate higher quality image results.Compared with three excellent methods on the Dunhuang grotto mural data set,and the repair results show that the PSNR score of the proposed algorithm on the test data set increases by 1.79%,and the SSIM score increases by 7.7%.The proposed repair model improves the repair accuracy of damaged murals and makes the repair results more accurate.

Key words: Expansion convolution, Generative adversarial networks, Mural inpainting, Perceptual loss

CLC Number: 

  • TP181
[1] CHEN Y,AI Y P,GUO H G.Inpainting algorithm for dun-huang mural based on improved curvature-driven diffusion mo-del[J].Journal of Computer-Aided Design & Computer Gra-phics,2020,32(5):787-796.
[2] ZHANG Z Y,SHUI W Y,ZHOU M Q,et al.Research on disease extraction and inpainting algorithm of digital grotto murals[J].Application Research of Computers,2021,38(8):1-5.
[3] BELHI A,BOURAS A,AL-ALI A K,et al.A machine learning framework for enhancing digital experiences in cultural heritage[J/OL].Journal of Enterprise Mnformation Management.https://doi.org/10.1108/JEIM-02-2020-0059.
[4] BERTALMIO M,SAPIRO G,CASELLES V,et a1.Image inpainting[C]//Proceedings of International Conference on Computer Graphics and Interactive Techniques.New Orleans,Louisiana,USA,2000:417-424.
[5] BERTALMIO M.Strong-continuation,contrast-invariant inpain-ting with a third order optimal PDE[J].IEEE Transaction on Image Processing,2006,15(7):1934-1938.
[6] CHAN T F,SHEN J H.Non-texture inpainting by curvature-driven diffusions(CDD)[J].Journal of Visual Communication and Image Representation,2001,12(4):436-449.
[7] CHAN R H,WEN Y W,YIP A M.A fast optimization transfer algorithm for image inpainting in wavelet domains[J].IEEE Transactions onImage Processing,2009,18(7):1467-1476.
[8] DRORI I,COHEN-OR D,YESHURUN H.Fragment-basedimage completion[C]//ACM S2GGRAPH.2003:303-312.
[9] CRIMINISI A,PEREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions On Image Processing.2004,13(9):25-29.
[10] XU Z,SUN J.Image inpainting by patch propagation usingpatch sparsity[J].IEEE Transactions onImage Processing,2010,19(5):1153-1165.
[11] YANG X P,WANG S W.Dunhuang mural inpainting in intricate disrepaired region based on improvement of priority algorithm[J].Journal of Computer-Aided Design & Computer Graphics,2011,23(2):284-289.
[12] JIAO L J,WANG W J,LI B J,et al.Wutai mountain mural inpainting based on improved block matching algorithm[J].Journal of Computer-Aided Design & Computer Graphics,2019,31(1):118-125.
[13] ZHAO L L,SHEN L,HONG R C.Survey on image inpainting research progress[J].Computer Science,2021,48(3):14-26.
[14] KÖHLER R,SCHULER C,SCHÖLKOPF B,et al.Mask-specific inpainting with deep neural networks[C]//German Confe-rence on Pattern Recognition.Cham:Springer,2014:523-534.
[15] XIE J,XU L,CHEN E.Image denoising and inpainting with deep neural networks[C]//Advances in Neural Information Processing Systems.2012:341-349.
[16] REN J S J,XU L,YAN Q,et al.Shepard convolutional neural networks[C]//Advances in Neural Information Processing Systems.2015:901-909.
[17] PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[18] LI Y,LIU S,YANG J,et al.Generative face completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3911-3919.
[19] IIZUKA S,SIMO-SERRA E,ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics,2017,36(4):1-14.
[20] YAN Z,LIX,LI M,et al.Shift-net:Image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision.2018:1-17.
[21] RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks[EB/OL].http://arxiv.Org/abs/1511.06434.
[22] ZOU Z,ZHAO P,ZHAO X F.Automatic segmentation,inpain-ting,and classification of defective patterns on ancient architecture using multiple deep learning algorithms[J/OL].Structural Contural & Health Monitoring.https://doi.org/10.1002/stc.2742.
[23] GOODFELLOW I,POUGET-ABADIE J,MIRZA M.Genera-tive Adversarial Networks[DB/OL].http://arxiv.Org/abs/1406.2661.
[24] MIRZA M,OSINDERO S.Conditional Generative AdversarialNets[DB/OL].http://arxiv.Org/abs/1411.1784.
[25] DENTON E,CHINTALA S,SZLAM A,et a1.Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks[DB/OL].http://arxiv.Org/abs/506.05751.
[26] LEI L,GUO D E,JIN F.Image Inpainting Algorithm Based on Conditional Generative Adversarial Network with Spectral Normalization[J].Computer Engineering,2021,47(1):230-238.
[27] ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein gan[J].arXiv:1701.07875,2017.
[28] CHEN Y,GAO Y.Image denoising via iterative diffusion me-thods combining two edge-indicators with adaptive thresholds[J].Multimedia Tools and Applications,2021,80(10):16027-16044.
[1] XU Guo-ning, CHEN Yi-peng, CHEN Yi-ming, CHEN Jin-yin, WEN Hao. Data Debiasing Method Based on Constrained Optimized Generative Adversarial Networks [J]. Computer Science, 2022, 49(6A): 184-190.
[2] GAO Zhi-yu, WANG Tian-jing, WANG Yue, SHEN Hang, BAI Guang-wei. Traffic Prediction Method for 5G Network Based on Generative Adversarial Network [J]. Computer Science, 2022, 49(4): 321-328.
[3] DOU Zhi, WANG Ning, WANG Shi-jie, WANG Zhi-hui, LI Hao-jie. Sketch Colorization Method with Drawing Prior [J]. Computer Science, 2022, 49(4): 195-202.
[4] LI Si-quan, WAN Yong-jing, JIANG Cui-ling. Multiple Fundamental Frequency Estimation Algorithm Based on Generative Adversarial Networks for Image Removal [J]. Computer Science, 2022, 49(3): 179-184.
[5] TAN Xin-yue, HE Xiao-hai, WANG Zheng-yong, LUO Xiao-dong, QING Lin-bo. Text-to-Image Generation Technology Based on Transformer Cross Attention [J]. Computer Science, 2022, 49(2): 107-115.
[6] ZHANG Wei-qi, TANG Yi-feng, LI Lin-yan, HU Fu-yuan. Image Stream From Paragraph Method Based on Scene Graph [J]. Computer Science, 2022, 49(1): 233-240.
[7] LIN Zhen-xian, ZHANG Meng-kai, WU Cheng-mao, ZHENG Xing-ning. Face Image Inpainting with Generative Adversarial Network [J]. Computer Science, 2021, 48(9): 174-180.
[8] XU Tao, TIAN Chong-yang, LIU Cai-hua. Deep Learning for Abnormal Crowd Behavior Detection:A Review [J]. Computer Science, 2021, 48(9): 125-134.
[9] WANG Le, YANG Xiao-min. Remote Sensing Image Pansharpening Feedback Network Based on Perceptual Loss [J]. Computer Science, 2021, 48(8): 91-98.
[10] PAN Xiao-qin, LU Tian-liang, DU Yan-hui, TONG Xin. Overview of Speech Synthesis and Voice Conversion Technology Based on Deep Learning [J]. Computer Science, 2021, 48(8): 200-208.
[11] YE Hong-liang, ZHU Wan-ning, HONG Lei. Music Style Transfer Method with Human Voice Based on CQT and Mel-spectrum [J]. Computer Science, 2021, 48(6A): 326-330.
[12] WANG Jian-ming, LI Xiang-feng, YE Lei, ZUO Dun-wen, ZHANG Li-ping. Medical Image Deblur Using Generative Adversarial Networks with Channel Attention [J]. Computer Science, 2021, 48(6A): 101-106.
[13] HU Yu-jie, CHANG Jian-hui, ZHANG Jian. Image Synthesis with Semantic Region Style Constraint [J]. Computer Science, 2021, 48(2): 134-141.
[14] YU Xiao-ming, HUANG Hua. Research on Application of Improved GAN Network in Generating Short Video [J]. Computer Science, 2021, 48(11A): 625-629.
[15] WEI Li-qi, ZHAO Zhi-hong, BAI Guang-wei, SHEN Hang. Location Privacy Game Mechanism Based on Generative Adversarial Networks [J]. Computer Science, 2021, 48(10): 266-271.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!