计算机科学 ›› 2022, Vol. 49 ›› Issue (6): 217-223.doi: 10.11896/jsjkx.210500105

• 计算机图形学&多媒体 • 上一篇    下一篇

基于特征感知的数字壁画复原方法

徐辉1,2, 康金梦1, 张加万1   

  1. 1 天津大学软件学院 天津 300072
    2 河南科技学院文化遗产数字传承研究中心 河南 新乡 453000
  • 收稿日期:2021-05-14 修回日期:2021-07-20 出版日期:2022-06-15 发布日期:2022-06-08
  • 通讯作者: 张加万(jwzhang@tju.edu.cn)
  • 作者简介:(huixu@tju.edu.cn)
  • 基金资助:
    国家重点研发计划(2019YFC1521200)

Digital Mural Inpainting Method Based on Feature Perception

XU Hui1,2, KANG Jin-meng1, ZHANG Jia-wan1   

  1. 1 School of Computer Software,Tianjin University,Tianjin 300072,China
    2 Digitization Technology Research Center for Cultural Heritage Conservation and Promotion,Henan Institute of Science and Technology,Xinxiang,Henan 453000,China
  • Received:2021-05-14 Revised:2021-07-20 Online:2022-06-15 Published:2022-06-08
  • About author:XU Hui,born in 1986,Ph.D,is a member of China Computer Federation.Her main research interests include image synthesis and digital conservation of cultural heritage.
    ZHANG Jia-wan,born in 1975,professor,Ph.D supervisor,is a member of China Computer Federation.His main research interests include image synthesis,visualization and visual analysis.
  • Supported by:
    National Key R & D Program of China(2019YFC1521200).

摘要: 敦煌壁画存在多种病害造成的不规则破损区域,运用数字修复的方式对其进行恢复,既不会对原始壁画造成损坏,又可以获得较好的修补效果。由于壁画修补问题中缺失的区域较大,不能用局部非语义的修补方法来实现。针对敦煌壁画缺损区域的修复问题,设计了基于生成对抗网络的图像修补方法,使用语义上合理的内容来渲染缺失区域的像素,实现非接触性壁画场景重建,从而提升壁画虚拟修复准确度。该算法在生成对抗神经网络的基础上引入感知损失函数,在生成模型中添加3层扩张卷积层来收集破损区域的图像特征,利用感知损失提升模型对高频纹理细节的修复能力,运用扩展卷积提取范围特征激励生成模型生成较高质量的图像结果。在敦煌壁画数据集上将所提方法与3种优秀方法进行了比较,修复结果显示所提算法在测试数据集上的PSNR评分提高了1.79%,SSIM评分提高了7.7%。所提修复模型提升了破损壁画的修复精度,使修复结果更加准确。

关键词: 壁画修复, 感知损失, 扩张卷积, 生成对抗网络

Abstract: There are irregular damaged areas caused by various diseases of grottoes in Dunhuang murals,digital restoration is used to restore the image of the Dunhuang grotto murals,which will not cause damage to the original murals,but also get a better repair effect.Because of the large missing area in the mural mending,it cannot be realized by local non-semantic repair methods.Aiming at the restoration of the defective area of Dunhuang grotto murals,this paper designs an image repair method based on the generation of confrontation network,and uses semantically reasonable content to render the pixels in the missing area to realize the reconstruction of non-contact mural scenes,improve the efficiency of mural virtual restoration and the accuracy of restoration.The algorithm introduces a perceptual-loss function on the basis of generating an adversarial neural network,adds a three-layer convolutional layer to the generation model to collect image features of damaged areas,uses the perceptual loss to improve the model’s ability to repair high-frequency texture details,and uses extended convolution to extract range features,so as to stimulate the generative model to generate higher quality image results.Compared with three excellent methods on the Dunhuang grotto mural data set,and the repair results show that the PSNR score of the proposed algorithm on the test data set increases by 1.79%,and the SSIM score increases by 7.7%.The proposed repair model improves the repair accuracy of damaged murals and makes the repair results more accurate.

Key words: Expansion convolution, Generative adversarial networks, Mural inpainting, Perceptual loss

中图分类号: 

  • TP181
[1] CHEN Y,AI Y P,GUO H G.Inpainting algorithm for dun-huang mural based on improved curvature-driven diffusion mo-del[J].Journal of Computer-Aided Design & Computer Gra-phics,2020,32(5):787-796.
[2] ZHANG Z Y,SHUI W Y,ZHOU M Q,et al.Research on disease extraction and inpainting algorithm of digital grotto murals[J].Application Research of Computers,2021,38(8):1-5.
[3] BELHI A,BOURAS A,AL-ALI A K,et al.A machine learning framework for enhancing digital experiences in cultural heritage[J/OL].Journal of Enterprise Mnformation Management.https://doi.org/10.1108/JEIM-02-2020-0059.
[4] BERTALMIO M,SAPIRO G,CASELLES V,et a1.Image inpainting[C]//Proceedings of International Conference on Computer Graphics and Interactive Techniques.New Orleans,Louisiana,USA,2000:417-424.
[5] BERTALMIO M.Strong-continuation,contrast-invariant inpain-ting with a third order optimal PDE[J].IEEE Transaction on Image Processing,2006,15(7):1934-1938.
[6] CHAN T F,SHEN J H.Non-texture inpainting by curvature-driven diffusions(CDD)[J].Journal of Visual Communication and Image Representation,2001,12(4):436-449.
[7] CHAN R H,WEN Y W,YIP A M.A fast optimization transfer algorithm for image inpainting in wavelet domains[J].IEEE Transactions onImage Processing,2009,18(7):1467-1476.
[8] DRORI I,COHEN-OR D,YESHURUN H.Fragment-basedimage completion[C]//ACM S2GGRAPH.2003:303-312.
[9] CRIMINISI A,PEREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions On Image Processing.2004,13(9):25-29.
[10] XU Z,SUN J.Image inpainting by patch propagation usingpatch sparsity[J].IEEE Transactions onImage Processing,2010,19(5):1153-1165.
[11] YANG X P,WANG S W.Dunhuang mural inpainting in intricate disrepaired region based on improvement of priority algorithm[J].Journal of Computer-Aided Design & Computer Graphics,2011,23(2):284-289.
[12] JIAO L J,WANG W J,LI B J,et al.Wutai mountain mural inpainting based on improved block matching algorithm[J].Journal of Computer-Aided Design & Computer Graphics,2019,31(1):118-125.
[13] ZHAO L L,SHEN L,HONG R C.Survey on image inpainting research progress[J].Computer Science,2021,48(3):14-26.
[14] KÖHLER R,SCHULER C,SCHÖLKOPF B,et al.Mask-specific inpainting with deep neural networks[C]//German Confe-rence on Pattern Recognition.Cham:Springer,2014:523-534.
[15] XIE J,XU L,CHEN E.Image denoising and inpainting with deep neural networks[C]//Advances in Neural Information Processing Systems.2012:341-349.
[16] REN J S J,XU L,YAN Q,et al.Shepard convolutional neural networks[C]//Advances in Neural Information Processing Systems.2015:901-909.
[17] PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[18] LI Y,LIU S,YANG J,et al.Generative face completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3911-3919.
[19] IIZUKA S,SIMO-SERRA E,ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics,2017,36(4):1-14.
[20] YAN Z,LIX,LI M,et al.Shift-net:Image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision.2018:1-17.
[21] RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks[EB/OL].http://arxiv.Org/abs/1511.06434.
[22] ZOU Z,ZHAO P,ZHAO X F.Automatic segmentation,inpain-ting,and classification of defective patterns on ancient architecture using multiple deep learning algorithms[J/OL].Structural Contural & Health Monitoring.https://doi.org/10.1002/stc.2742.
[23] GOODFELLOW I,POUGET-ABADIE J,MIRZA M.Genera-tive Adversarial Networks[DB/OL].http://arxiv.Org/abs/1406.2661.
[24] MIRZA M,OSINDERO S.Conditional Generative AdversarialNets[DB/OL].http://arxiv.Org/abs/1411.1784.
[25] DENTON E,CHINTALA S,SZLAM A,et a1.Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks[DB/OL].http://arxiv.Org/abs/506.05751.
[26] LEI L,GUO D E,JIN F.Image Inpainting Algorithm Based on Conditional Generative Adversarial Network with Spectral Normalization[J].Computer Engineering,2021,47(1):230-238.
[27] ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein gan[J].arXiv:1701.07875,2017.
[28] CHEN Y,GAO Y.Image denoising via iterative diffusion me-thods combining two edge-indicators with adaptive thresholds[J].Multimedia Tools and Applications,2021,80(10):16027-16044.
[1] 张佳, 董守斌.
基于评论方面级用户偏好迁移的跨领域推荐算法
Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer
计算机科学, 2022, 49(9): 41-47. https://doi.org/10.11896/jsjkx.220200131
[2] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[3] 戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮.
基于DNGAN的磁共振图像超分辨率重建算法
Super-resolution Reconstruction of MRI Based on DNGAN
计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105
[4] 尹文兵, 高戈, 曾邦, 王霄, 陈怡.
基于时频域生成对抗网络的语音增强算法
Speech Enhancement Based on Time-Frequency Domain GAN
计算机科学, 2022, 49(6): 187-192. https://doi.org/10.11896/jsjkx.210500114
[5] 高志宇, 王天荆, 汪悦, 沈航, 白光伟.
基于生成对抗网络的5G网络流量预测方法
Traffic Prediction Method for 5G Network Based on Generative Adversarial Network
计算机科学, 2022, 49(4): 321-328. https://doi.org/10.11896/jsjkx.210300240
[6] 黎思泉, 万永菁, 蒋翠玲.
基于生成对抗网络去影像的多基频估计算法
Multiple Fundamental Frequency Estimation Algorithm Based on Generative Adversarial Networks for Image Removal
计算机科学, 2022, 49(3): 179-184. https://doi.org/10.11896/jsjkx.201200081
[7] 李建, 郭延明, 于天元, 武与伦, 王翔汉, 老松杨.
基于生成对抗网络的多目标类别对抗样本生成算法
Multi-target Category Adversarial Example Generating Algorithm Based on GAN
计算机科学, 2022, 49(2): 83-91. https://doi.org/10.11896/jsjkx.210800130
[8] 谈馨悦, 何小海, 王正勇, 罗晓东, 卿粼波.
基于Transformer交叉注意力的文本生成图像技术
Text-to-Image Generation Technology Based on Transformer Cross Attention
计算机科学, 2022, 49(2): 107-115. https://doi.org/10.11896/jsjkx.210600085
[9] 陈贵强, 何军.
自然场景下遥感图像超分辨率重建算法研究
Study on Super-resolution Reconstruction Algorithm of Remote Sensing Images in Natural Scene
计算机科学, 2022, 49(2): 116-122. https://doi.org/10.11896/jsjkx.210700095
[10] 石达, 芦天亮, 杜彦辉, 张建岭, 暴雨轩.
基于改进CycleGAN的人脸性别伪造图像生成模型
Generation Model of Gender-forged Face Image Based on Improved CycleGAN
计算机科学, 2022, 49(2): 31-39. https://doi.org/10.11896/jsjkx.210600012
[11] 唐雨潇, 王斌君.
基于深度生成模型的人脸编辑研究进展
Research Progress of Face Editing Based on Deep Generative Model
计算机科学, 2022, 49(2): 51-61. https://doi.org/10.11896/jsjkx.210400108
[12] 张玮琪, 汤轶丰, 李林燕, 胡伏原.
基于场景图的段落生成序列图像方法
Image Stream From Paragraph Method Based on Scene Graph
计算机科学, 2022, 49(1): 233-240. https://doi.org/10.11896/jsjkx.201100207
[13] 蒋宗礼, 樊珂, 张津丽.
基于生成对抗网络和元路径的异质网络表示学习
Generative Adversarial Network and Meta-path Based Heterogeneous Network Representation Learning
计算机科学, 2022, 49(1): 133-139. https://doi.org/10.11896/jsjkx.201000179
[14] 林椹尠, 张梦凯, 吴成茂, 郑兴宁.
利用生成对抗网络的人脸图像分步补全法
Face Image Inpainting with Generative Adversarial Network
计算机科学, 2021, 48(9): 174-180. https://doi.org/10.11896/jsjkx.200800014
[15] 刘立波, 苟婷婷.
融合深度典型相关分析和对抗学习的跨模态检索
Cross-modal Retrieval Combining Deep Canonical Correlation Analysis and Adversarial Learning
计算机科学, 2021, 48(9): 200-207. https://doi.org/10.11896/jsjkx.200600119
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!