Computer Science ›› 2024, Vol. 51 ›› Issue (6A): 230400083-9.doi: 10.11896/jsjkx.230400083

• Image Processing & Multimedia Technolog • Previous Articles     Next Articles

Mural Inpainting Based on Fast Fourier Convolution and Feature Pruning Coordinate Attention

ZHANG Le, YU Ying, GE Hao   

  1. School of Information,Yunnan University,Kunming 650500,China
  • Published:2024-06-06
  • About author:ZHANG Le,born in 2000,postgraduate.His main research interests include image inpainting and deep learning.
    YU Ying,born in 1977,Ph.D,associate professor.His main research interests include image and vision,artificial neural network.
  • Supported by:
    National Natural Science Foundation of China(62166048,61263048) and Yunnan Province Applied Basic Research Project(2018FB102).

Abstract: A proposed solution to the problem of high manual inpainting costs for ancient murals that have undergone varying degrees of natural weathering resulting in cracks,peeling,and other damage is to use a generative adversarial network with a framework based on fast Fourier convolution and coordinate attention.Most existing methods for mural inpainting have complex frameworks that consume a lot of computing power,and produce results that are inaccurate and of low quality.The proposed method takes the damaged mural image and mask as inputs to the network.They are then passed through an encoder and a residual module for feature inference to determine the reasonable content of the damaged area.During training,a specific discriminator that is used for inpainting tasks conducts adversarial training.Eventually,the desired inpainting effect is achieved.The feature inference portion of the proposed model consists of a residual block containing gate-controlled residual connections,six fast Fourier convolution modules,and an improved coordinate attention module for feature pruning.It has a large receptive field and the ability to extract rich features,which can solve the problem of poor inpainting results associated with current methods.Experimental results on a self-made dataset show that the proposed algorithm not only has a simpler structure but also outperforms several classic inpainting methods.Therefore,it can be applied to the inpainting of ancient murals and can save a significant amount of manual labor costs.

Key words: Mural inpainting, Attention mechanism, Residual network, Generative adversarial network, Deep learning

CLC Number: 

  • TP391
[1]CAO J F,ZHANG Z B,ZHAO A D.Application of Enhanced Consistent Generative Adversarial Network in Mural Repairing[J].Journal of Computer-Aided Design & Computer Graphics,2020,32(8):1315-1323.
[2]SHEN J N,WANG H Q,WU M.Tang Dynasty Tomb Murals Inpainting Algorithm of MCA Decomposition[J].Journal of Frontiers of Computer Science and Technology,2017,11(11):1826.
[3]HUANG W,WANG S W,YANG X P.Inpainting Method of Dunhuang Murals Based on Image Decomposition[J].Journal of Shandong University(Engineering Science),2010,40(2):24-27.
[4]YANG X P,WANG S W.Restoration of Dunhuang MuralsBased on Markov Sampling[J].Journal of Computer Applications.2010(7):1835-1837.
[5]JIANG J,ZHUO G,QI X W.Research on Tibetan Muralinpainting Technology Based on Richardson Lucy Algorithm[J].China Computer and Communication,2022,34(10):3.
[6]JIA R.The Application of Digital inpainting Technology in Tang Tomb Murals[D].Xi’an:Xi’an University of Architecture and Technology,2014.
[7]YANG T,WANG S S,PEN H B.Automatic recognition and repair of cracks in mural images based on improved SOM[J].Journalof Tianjin University:Science and Technology,2020,53(9):932-938.
[8]CHEN Y,TAO M F,CHEN J.Mural inpainting based on RPCAdecomposition of block nuclear norm and entropy weighted clustering sparse representation[J].Journal of Harbin Institute of Technology,2021,53(8):72-80.
[9]LI Q Q,WANG H,ZOU Q.A Murals Inpainting Algorithm Based on Sparse Representation Model[J].Geomatics and Information Science of Wuhan University,2018,43(12):1847-1853.
[10]CAO Y.Research on Digital Restoration Technology of Dunhuang Fresco Based on Generalized Regression Neural Network[D].Gansu:Northwest Normal University,2017.
[11]LIANG L.The Application of inpainting Algorithm Based on Samples in Tang Tomb Murals[D].Xi’an:Xi’an University of Architecture and Technology,2013.
[12]SUVOROV R,LOGACHEVA E,MASHIKHIN A,et al.Resolution-robust large mask inpainting with fourier convolutions[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2022:2149-2159.
[13]ZENG Y,FU J,CHAO H,et al.Aggregated contextual transformations for high-resolution image inpainting[J].IEEE Transactions on Visualization and Computer Graphics,2023,29(7):3266-3280.
[14]LI J,WANG N,ZHANG L,et al.Recurrent feature reasoning for image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7760-7768.
[15]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018:85-100.
[16]WOO S,PARK J,LEE J Y,et al.Cbam:Convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018:3-19.
[17]NAZERI K,NG E,JOSEPH T,et al.Edgeconnect:Generativeimage inpainting with adversarial edge learning[J].arXiv:1901.00212,2019.
[18]HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[19]WANG Q,WU B,ZHU P,et al.ECA-Net:Efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:11534-11542.
[20]HOU Q,ZHOU D,FENG J.Coordinate attention for efficient mobile network design[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:13713-13722.
[21]WANG N,ZHANG Y,ZHANG L.Dynamic selection networkfor image inpainting[J].IEEE Transactions on Image Processing,2021,30:1784-1798.
[22]UDDIN S M N,JUNG Y J.Global and local attention-based free-form image inpainting[J].Sensors,2020,20(11):3204.
[23]JOHNSON J,ALAHI A,FEI-FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//Computer Vision-ECCV 2016:14th European Conference,Amsterdam,The Netherlands,Part II 14.Springer International Publishing,2016:694-711.
[24]GATYS L A,ECKER A S,BETHGE M.Image style transferusing convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2414-2423.
[25]ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:1125-1134.
[26]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[27]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial networks[J].Communications of the ACM,2020,63(11):139-144.
[28]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[1] WU Chunming, WANG Tiaojun. Study on Defect Detection Algorithm of Transmission Line in Complex Background [J]. Computer Science, 2024, 51(6A): 230500178-6.
[2] LYU Yiming, WANG Jiyang. Iron Ore Image Classification Method Based on Improved Efficientnetv2 [J]. Computer Science, 2024, 51(6A): 230600212-6.
[3] YANG Xiuzhang, WU Shuai, REN Tianshu, LIAO Wenjing, XIANG Meiyu, YU Xiaomin, LIU Jianyi, CHEN Dengjian. Complex Environment License Plate Recognition Algorithm Based on Improved Image Enhancement and CNN [J]. Computer Science, 2024, 51(6A): 220200162-7.
[4] SONG Zhen, WANG Jiqiang, HOU Moyu, ZHAO Lin. Conveyor Belt Defect Detection Network Combining Attention Mechanism with Line Laser Assistance [J]. Computer Science, 2024, 51(6A): 230800115-6.
[5] WU Chunming, LIU Yali. Method for Lung Nodule Detection on CT Images Using Improved YOLOv5 [J]. Computer Science, 2024, 51(6A): 230500019-6.
[6] XIAO Yahui, ZHANG Zili, HU Xinrong, PENG Tao, ZHANG Jun. Clothing Image Segmentation Method Based on Deeplabv3+ Fused with Attention Mechanism [J]. Computer Science, 2024, 51(6A): 230900153-7.
[7] LANG Lang, CHEN Xiaoqin, LIU Sha, ZHOU Qiang. Detection of Pitting Defects on the Surface of Ball Screw Drive Based on Improved Deeplabv3+ Algorithm [J]. Computer Science, 2024, 51(6A): 240200058-6.
[8] ZHUO Peiyan, ZHANG Yaona, LIU Wei, LIU Zijin, SONG You. CTGANBoost:Credit Fraud Detection Based on CTGAN and Boosting [J]. Computer Science, 2024, 51(6A): 230600199-7.
[9] YIN Xudong, CHEN Junyang, ZHOU Bo. Study on Industrial Defect Augmentation Data Filtering Based on OOD Scores [J]. Computer Science, 2024, 51(6A): 230700111-7.
[10] QIAO Hong, XING Hongjie. Attention-based Multi-scale Distillation Anomaly Detection [J]. Computer Science, 2024, 51(6A): 230300223-11.
[11] KANG Zhiyong, LI Bicheng, LIN Huang. User Interest Recognition Method Incorporating Category Labels and Topic Information [J]. Computer Science, 2024, 51(6A): 230500169-8.
[12] SI Jia, LIANG Jianfeng, XIE Shuo, DENG Yingjun. Research Progress of Anomaly Detection in IaaS Cloud Operation Driven by Deep Learning [J]. Computer Science, 2024, 51(6A): 230400016-8.
[13] DUAN Pengsong, DIAO Xianguang, ZHANG Dalong, CAO Yangjie, LIU Guangyi, KONG Jinsheng. WiCare:Non-contact Fall Monitoring Model for Elderly in Toilet [J]. Computer Science, 2024, 51(6A): 230700044-8.
[14] WANG Yingjie, ZHANG Chengye, BAI Fengbo, WANG Zumin. Named Entity Recognition Approach of Judicial Documents Based on Transformer [J]. Computer Science, 2024, 51(6A): 230500164-9.
[15] ZHU Yuliang, LIU Juntao, RAO Ziyun, ZHANG Yi, CAO Wanhua. Knowledge Reasoning Model Combining HousE with Attention Mechanism [J]. Computer Science, 2024, 51(6A): 230600209-8.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!