Computer Science ›› 2023, Vol. 50 ›› Issue (6A): 220600129-8.doi: 10.11896/jsjkx.220600129

• Image Processing & Multimedia Technology • Previous Articles     Next Articles

Two-stage Method for Restoration of Heritage Images Based on Muti-scale Attention Mechanism

LIU Haowei, YAO Jingchi, LIU Bo, BI Xiuli, XIAO Bin   

  1. Chongqing Key Laboratory of Image Cognition,Chongqing 400065,China
  • Online:2023-06-10 Published:2023-06-12
  • About author:LIU Haowei,born in 2001,undergra-duate student.His main research intere-sts include image processing and deep learning. LIU Bo,born in 1987,Ph.D,lecturer,is a member of China Computer Federation.His main research interests include multimedia forensics and computer vision.
  • Supported by:
    National-level Student Innovation and Entrepreneurship Training Program(202110617006).

Abstract: The use of virtual technology is important for the restoration of relics,which are often damaged by improper preservation or physical restoration methods.Existing traditional image restoration techniques and deep learning-based restoration methods are mainly suitable for images with simple structural textures,small damaged areas,or natural images with regular damage,and cannot be directly applied to heritage images.Using landscape painting image restoration as an example,a two-stage method for restoration of heritage images based on a multi-scale attention mechanism is proposed in this paper to address the problems of complex structural textures,discreet colouring and small size of existing datasets of heritage images.The method firstly performs coarse restoration of the overall structure and base tones of the image based on the global attention mechanism,then performs local fine restoration of small structures and fine textures of the image using the local attention mechanism and the residual module,as well as global fine restoration of large structures and textures using the contextual attention mechanism on the result of coarse restoration to borrow information accurately at a distance.Finally,the local and global fine restoration results are fused to achieve the restoration of heritage images.The proposed method has the advantage of improving the peak signal-to-noise ratio by 3.76 dB and the structural similarity by 0.034 compared with the comparative methods on average.Both the subjective and objective analysis of the experimental results show that the method has some advantages in semantic rationality,information accuracy and visual naturalness compared with the existing methods,and has a high potential for application in the field of heritage restoration.

Key words: Heritage image, Image restoration, Deep learning, Two stage model, Muti-scale attention mechanism

CLC Number: 

  • TP391
[1]MA G C.Analysis of the application of digital image restoration technology in the conservation of cultural relics[J].Cultural Heritage Identification and Appreciation,2021,214(19):106-108.
[2]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.2000.
[3]BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:A randomized correspondence algorithm for structural image editing[J].ACM Trans Graph,2009,28(3):24.
[4]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200.
[5]HUANG J B,KANG S B,AHUJA N,et al.Image completionusing planar structure guidance[J].ACM Transactions on graphics(TOG),2014,33(4):1-10.
[6]STARCK J L,ELAD M,DONOHO D L.Image decompositionvia the combination of sparse representations and a variational approach[J].IEEE Transactions on Image Processing,2005,14(10):1570-1582.
[7]MAIRAL J,ELAD M,SAPIRO G.Sparse representation forcolor image restoration[J].IEEE Transactions on Image Processing,2007,17(1):53-69.
[8]YAN Z,LI X,LI M,et al.Shift-net:Image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018.
[9]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018.
[10]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016.
[11]LIU H,JIANG B,XIAO Y,et al.Coherent semantic attention for image inpainting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019.
[12]YI Z,TANG Q,AZIZI S,et al.Contextual residual aggregation for ultra high-resolution image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020.
[13]NAZERI K,NG E,JOSEPH T,et al.Edgeconnect:Structureguided image inpainting using edge prediction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.2019.
[14]YU J,LIN Z,YANG J,et al.Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019.
[15]YU J,LIN Z,YANG J,et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[16]IIZUKA S,SIMO-SERRA E,ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics(ToG),2017,36(4):1-14.
[17]REN Y,YU X,ZHANG R,et al.Structureflow:Image inpain-ting via structure-aware appearance flow[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019.
[18]WANG Y,CHEN Y C,TAO X,et al.Vcnet:A robust approach to blind image inpainting[C]//Proceedings of the European Conference on Computer Vision.2020.
[19]ZENG Y,LIN Z,LU H,et al.Cr-fill:Generative image inpain-ting with auxiliary contextual reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021.
[20]ZENG Y,LIN Z,YANG J,et al.High-resolution image inpain-ting with iterative confidence feedback and guided upsampling[C]//Proceedings of the European Conference on Computer Vision.2020.
[21]GUO Z,CHEN Z,YU T,et al.Progressive image inpaintingwith full-resolution residual network[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019.
[22]LI J,HE F,ZHANG L,et al.Progressive reconstruction of vi-sual structure for image inpainting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019.
[23]LI J,WANG N,ZHANG L,et al.Recurrent feature reasoning for image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020.
[24]ZHANG H,HU Z,LUO C,et al.Semantic image inpaintingwith progressive generative networks[C]//Proceedings of the 26th ACM International Conference on Multimedia.2018.
[25]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention.2015.
[26]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017.
[27]RADFORD A,METZ L,CHINTALA S.Unsupervised repre-sentation learning with deep convolutional generative adversarial networks[J].arXiv:151106434,2015,
[28]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Ge-nerative adversarial networks[J].Communications of the ACM,2020,63(11):139-144.
[29]LIU H,JIANG B,SONG Y,et al.Rethinking image inpainting via a mutual encoder-decoder with feature equalizations[C]//Proceedings of the European Conference on Computer Vision.2020.
[30]SHARAN L,ROSENHOLTZ R,ADELSON E H.Accuracy and speed of material categorization in real-world images[J].Journal of Vision,2014,14(9):12.
[31]FUKUSHIMA K,MIYAKE S.Neocognitron:A self-organizing neural network model for a mechanism of visual pattern recognition[M].Competition and Cooperation in Neural Nets.Sprin-ger.1982:267-85.
[32]XUE A.End-to-end chinese landscape painting creation using generative adversarial networks[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2021.
[33]HORE A,ZIOU D.Image quality metrics:PSNR vs. SSIM[C]//Proceedings of the 2010 20th International Conference on Pattern Recognition.2010.
[34]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[1] ZHANG Yian, YANG Ying, REN Gang, WANG Gang. Study on Multimodal Online Reviews Helpfulness Prediction Based on Attention Mechanism [J]. Computer Science, 2023, 50(8): 37-44.
[2] SONG Xinyang, YAN Zhiyuan, SUN Muyi, DAI Linlin, LI Qi, SUN Zhenan. Review of Talking Face Generation [J]. Computer Science, 2023, 50(8): 68-78.
[3] WANG Xu, WU Yanxia, ZHANG Xue, HONG Ruize, LI Guangsheng. Survey of Rotating Object Detection Research in Computer Vision [J]. Computer Science, 2023, 50(8): 79-92.
[4] ZHOU Ziyi, XIONG Hailing. Image Captioning Optimization Strategy Based on Deep Learning [J]. Computer Science, 2023, 50(8): 99-110.
[5] ZHANG Xiao, DONG Hongbin. Lightweight Multi-view Stereo Integrating Coarse Cost Volume and Bilateral Grid [J]. Computer Science, 2023, 50(8): 125-132.
[6] WANG Yu, WANG Zuchao, PAN Rui. Survey of DGA Domain Name Detection Based on Character Feature [J]. Computer Science, 2023, 50(8): 251-259.
[7] LI Kun, GUO Wei, ZHANG Fan, DU Jiayu, YANG Meiyue. Adversarial Malware Generation Method Based on Genetic Algorithm [J]. Computer Science, 2023, 50(7): 325-331.
[8] WANG Mingxia, XIONG Yun. Disease Diagnosis Prediction Algorithm Based on Contrastive Learning [J]. Computer Science, 2023, 50(7): 46-52.
[9] SHEN Zhehui, WANG Kailai, KONG Xiangjie. Exploring Station Spatio-Temporal Mobility Pattern:A Short and Long-term Traffic Prediction Framework [J]. Computer Science, 2023, 50(7): 98-106.
[10] HUO Weile, JING Tao, REN Shuang. Review of 3D Object Detection for Autonomous Driving [J]. Computer Science, 2023, 50(7): 107-118.
[11] ZHOU Bo, JIANG Peifeng, DUAN Chang, LUO Yuetong. Study on Single Background Object Detection Oriented Improved-RetinaNet Model and Its Application [J]. Computer Science, 2023, 50(7): 137-142.
[12] MAO Huihui, ZHAO Xiaole, DU Shengdong, TENG Fei, LI Tianrui. Short-term Subway Passenger Flow Forecasting Based on Graphical Embedding of Temporal Knowledge [J]. Computer Science, 2023, 50(7): 213-220.
[13] LI Yuqiang, LI Linfeng, ZHU Hao, HOU Mengshu. Deep Learning-based Algorithm for Active IPv6 Address Prediction [J]. Computer Science, 2023, 50(7): 261-269.
[14] GAO Xiang, TANG Jiqiang, ZHU Junwu, LIANG Mingxuan, LI Yang. Study on Named Entity Recognition Method Based on Knowledge Graph Enhancement [J]. Computer Science, 2023, 50(6A): 220700153-6.
[15] ZENG Wu, MAO Guojun. Few-shot Learning Method Based on Multi-graph Feature Aggregation [J]. Computer Science, 2023, 50(6A): 220400029-10.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!