计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (6A): 230400083-9.doi: 10.11896/jsjkx.230400083

• 图像处理&多媒体技术 • 上一篇    下一篇

基于快速傅里叶卷积与特征修剪坐标注意力的壁画修复

张乐, 余映, 革浩   

  1. 云南大学信息学院 昆明 650500
  • 发布日期:2024-06-06
  • 通讯作者: 余映(yuying.mail@163.com)
  • 作者简介:(1020423507@qq.com)
  • 基金资助:
    国家自然科学基金(62166048,61263048);云南省应用基础研究计划项目(2018FB102)

Mural Inpainting Based on Fast Fourier Convolution and Feature Pruning Coordinate Attention

ZHANG Le, YU Ying, GE Hao   

  1. School of Information,Yunnan University,Kunming 650500,China
  • Published:2024-06-06
  • About author:ZHANG Le,born in 2000,postgraduate.His main research interests include image inpainting and deep learning.
    YU Ying,born in 1977,Ph.D,associate professor.His main research interests include image and vision,artificial neural network.
  • Supported by:
    National Natural Science Foundation of China(62166048,61263048) and Yunnan Province Applied Basic Research Project(2018FB102).

PDF (PC)

306

摘要： 针对现存古代壁画长时间自然风化引起的不同程度的裂缝、脱落等病害,人工修复成本过高,而目前已有的壁画修复方法大多都存在框架复杂、耗费算力大,并且修复色彩不够准确和质量不够高等问题,提出了一种以快速傅里叶卷积和坐标注意力为框架的生成对抗网络用于修复工作。该方法将待修复壁画图像和掩码输入该网络,经编码器后传入用于特征推理的残差模块以推理出待修复区域的合理内容;训练过程中由特定的用于修复任务的鉴别器进行对抗训练,最终达到修复效果。所提模型中的特征推理部分为一个包含门控残差连接、6个快速傅里叶卷积模块和改进的特征修剪坐标注意力模块的残差块,具有较大的感受野和提取丰富特征的能力,可解决当前方法所存在的修复结果不佳的问题。在自制数据集下进行实验,与现有几种经典的修复方法进行对比的结果表明,所提算法不仅结构简单,还有着更优秀的修复能力,可应用于古代壁画修复工作,可节省大量的人工成本。

关键词: 古代壁画修复, 注意力机制, 残差网络, 生成对抗网络, 深度学习

Abstract: A proposed solution to the problem of high manual inpainting costs for ancient murals that have undergone varying degrees of natural weathering resulting in cracks,peeling,and other damage is to use a generative adversarial network with a framework based on fast Fourier convolution and coordinate attention.Most existing methods for mural inpainting have complex frameworks that consume a lot of computing power,and produce results that are inaccurate and of low quality.The proposed method takes the damaged mural image and mask as inputs to the network.They are then passed through an encoder and a residual module for feature inference to determine the reasonable content of the damaged area.During training,a specific discriminator that is used for inpainting tasks conducts adversarial training.Eventually,the desired inpainting effect is achieved.The feature inference portion of the proposed model consists of a residual block containing gate-controlled residual connections,six fast Fourier convolution modules,and an improved coordinate attention module for feature pruning.It has a large receptive field and the ability to extract rich features,which can solve the problem of poor inpainting results associated with current methods.Experimental results on a self-made dataset show that the proposed algorithm not only has a simpler structure but also outperforms several classic inpainting methods.Therefore,it can be applied to the inpainting of ancient murals and can save a significant amount of manual labor costs.

Key words: Mural inpainting, Attention mechanism, Residual network, Generative adversarial network, Deep learning

中图分类号: 

  • TP391
[1]CAO J F,ZHANG Z B,ZHAO A D.Application of Enhanced Consistent Generative Adversarial Network in Mural Repairing[J].Journal of Computer-Aided Design & Computer Graphics,2020,32(8):1315-1323.
[2]SHEN J N,WANG H Q,WU M.Tang Dynasty Tomb Murals Inpainting Algorithm of MCA Decomposition[J].Journal of Frontiers of Computer Science and Technology,2017,11(11):1826.
[3]HUANG W,WANG S W,YANG X P.Inpainting Method of Dunhuang Murals Based on Image Decomposition[J].Journal of Shandong University(Engineering Science),2010,40(2):24-27.
[4]YANG X P,WANG S W.Restoration of Dunhuang MuralsBased on Markov Sampling[J].Journal of Computer Applications.2010(7):1835-1837.
[5]JIANG J,ZHUO G,QI X W.Research on Tibetan Muralinpainting Technology Based on Richardson Lucy Algorithm[J].China Computer and Communication,2022,34(10):3.
[6]JIA R.The Application of Digital inpainting Technology in Tang Tomb Murals[D].Xi’an:Xi’an University of Architecture and Technology,2014.
[7]YANG T,WANG S S,PEN H B.Automatic recognition and repair of cracks in mural images based on improved SOM[J].Journalof Tianjin University:Science and Technology,2020,53(9):932-938.
[8]CHEN Y,TAO M F,CHEN J.Mural inpainting based on RPCAdecomposition of block nuclear norm and entropy weighted clustering sparse representation[J].Journal of Harbin Institute of Technology,2021,53(8):72-80.
[9]LI Q Q,WANG H,ZOU Q.A Murals Inpainting Algorithm Based on Sparse Representation Model[J].Geomatics and Information Science of Wuhan University,2018,43(12):1847-1853.
[10]CAO Y.Research on Digital Restoration Technology of Dunhuang Fresco Based on Generalized Regression Neural Network[D].Gansu:Northwest Normal University,2017.
[11]LIANG L.The Application of inpainting Algorithm Based on Samples in Tang Tomb Murals[D].Xi’an:Xi’an University of Architecture and Technology,2013.
[12]SUVOROV R,LOGACHEVA E,MASHIKHIN A,et al.Resolution-robust large mask inpainting with fourier convolutions[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2022:2149-2159.
[13]ZENG Y,FU J,CHAO H,et al.Aggregated contextual transformations for high-resolution image inpainting[J].IEEE Transactions on Visualization and Computer Graphics,2023,29(7):3266-3280.
[14]LI J,WANG N,ZHANG L,et al.Recurrent feature reasoning for image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7760-7768.
[15]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018:85-100.
[16]WOO S,PARK J,LEE J Y,et al.Cbam:Convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018:3-19.
[17]NAZERI K,NG E,JOSEPH T,et al.Edgeconnect:Generativeimage inpainting with adversarial edge learning[J].arXiv:1901.00212,2019.
[18]HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[19]WANG Q,WU B,ZHU P,et al.ECA-Net:Efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:11534-11542.
[20]HOU Q,ZHOU D,FENG J.Coordinate attention for efficient mobile network design[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:13713-13722.
[21]WANG N,ZHANG Y,ZHANG L.Dynamic selection networkfor image inpainting[J].IEEE Transactions on Image Processing,2021,30:1784-1798.
[22]UDDIN S M N,JUNG Y J.Global and local attention-based free-form image inpainting[J].Sensors,2020,20(11):3204.
[23]JOHNSON J,ALAHI A,FEI-FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//Computer Vision－ECCV 2016:14th European Conference,Amsterdam,The Netherlands,Part II 14.Springer International Publishing,2016:694-711.
[24]GATYS L A,ECKER A S,BETHGE M.Image style transferusing convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2414-2423.
[25]ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:1125-1134.
[26]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[27]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial networks[J].Communications of the ACM,2020,63(11):139-144.
[28]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发