计算机科学

计算机科学 ›› 2023, Vol. 50 ›› Issue (6A): 220600129-8.doi: 10.11896/jsjkx.220600129

• 图像处理&多媒体技术 • 上一篇    下一篇

基于多尺度注意力机制的两阶段文物图像修复方法

刘浩威, 姚镜池, 刘波, 毕秀丽, 肖斌   

  1. 图像认知重庆市重点实验室 重庆 400065
  • 出版日期:2023-06-10 发布日期:2023-06-12
  • 通讯作者: 刘波(boliu@cqupt.edu.cn )
  • 作者简介:(liu_haow@163.com)
  • 基金资助:
    国家级大学生创新创业训练计划支持项目(202110617006)

Two-stage Method for Restoration of Heritage Images Based on Muti-scale Attention Mechanism

LIU Haowei, YAO Jingchi, LIU Bo, BI Xiuli, XIAO Bin   

  1. Chongqing Key Laboratory of Image Cognition,Chongqing 400065,China
  • Online:2023-06-10 Published:2023-06-12
  • About author:LIU Haowei,born in 2001,undergra-duate student.His main research intere-sts include image processing and deep learning. LIU Bo,born in 1987,Ph.D,lecturer,is a member of China Computer Federation.His main research interests include multimedia forensics and computer vision.
  • Supported by:
    National-level Student Innovation and Entrepreneurship Training Program(202110617006).

PDF (PC)

833

摘要： 文物常因保存或物理修复手段不当而受到损坏,使用虚拟技术对其进行修复很重要,而现有传统图像修复技术和基于深度学习的修复方法主要针对结构纹理简单、破损区域较小的图像或是破损区域规则的自然图像,无法直接应用于文物图像。针对文物图像结构纹理复杂、破损区域不规则及现存文物图像数据集较小等问题,以山水画图像修复为例提出了一种基于多尺度注意力机制的两阶段文物图像修复方法。首先基于全局注意力机制对文物图像的整体结构和基础色调进行粗粒度修复,然后使用局部注意力机制和残差模块对文物图像的小型结构和细节纹理进行局部细粒度修复,并在粗粒度修复的结果上使用上下文注意力机制从文物图像远距离精确借用信息,对图像的大型结构和纹理进行全局细粒度修复,最后将局部和全局的修复结果进行特征融合,实现文物图像的修复。针对文物图像特殊的破损类型,修复的文物图像伪迹较少,色彩均匀,结构纹理清晰,相比对比方法,在峰值信噪比上平均提高了3.76dB,在结构相似性上平均提高了0.034。实验结果的主观和客观分析表明,与主流图像修复方法相比,在语义合理性、信息准确性和视觉自然性上都具有一定优势,在文物修复领域有较大应用价值。

关键词: 文物图像, 图像修复, 深度学习, 两阶段模型, 多尺度注意力机制

Abstract: The use of virtual technology is important for the restoration of relics,which are often damaged by improper preservation or physical restoration methods.Existing traditional image restoration techniques and deep learning-based restoration methods are mainly suitable for images with simple structural textures,small damaged areas,or natural images with regular damage,and cannot be directly applied to heritage images.Using landscape painting image restoration as an example,a two-stage method for restoration of heritage images based on a multi-scale attention mechanism is proposed in this paper to address the problems of complex structural textures,discreet colouring and small size of existing datasets of heritage images.The method firstly performs coarse restoration of the overall structure and base tones of the image based on the global attention mechanism,then performs local fine restoration of small structures and fine textures of the image using the local attention mechanism and the residual module,as well as global fine restoration of large structures and textures using the contextual attention mechanism on the result of coarse restoration to borrow information accurately at a distance.Finally,the local and global fine restoration results are fused to achieve the restoration of heritage images.The proposed method has the advantage of improving the peak signal-to-noise ratio by 3.76 dB and the structural similarity by 0.034 compared with the comparative methods on average.Both the subjective and objective analysis of the experimental results show that the method has some advantages in semantic rationality,information accuracy and visual naturalness compared with the existing methods,and has a high potential for application in the field of heritage restoration.

Key words: Heritage image, Image restoration, Deep learning, Two stage model, Muti-scale attention mechanism

中图分类号: 

  • TP391
[1]MA G C.Analysis of the application of digital image restoration technology in the conservation of cultural relics[J].Cultural Heritage Identification and Appreciation,2021,214(19):106-108.
[2]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.2000.
[3]BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:A randomized correspondence algorithm for structural image editing[J].ACM Trans Graph,2009,28(3):24.
[4]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200.
[5]HUANG J B,KANG S B,AHUJA N,et al.Image completionusing planar structure guidance[J].ACM Transactions on graphics(TOG),2014,33(4):1-10.
[6]STARCK J L,ELAD M,DONOHO D L.Image decompositionvia the combination of sparse representations and a variational approach[J].IEEE Transactions on Image Processing,2005,14(10):1570-1582.
[7]MAIRAL J,ELAD M,SAPIRO G.Sparse representation forcolor image restoration[J].IEEE Transactions on Image Processing,2007,17(1):53-69.
[8]YAN Z,LI X,LI M,et al.Shift-net:Image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018.
[9]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018.
[10]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016.
[11]LIU H,JIANG B,XIAO Y,et al.Coherent semantic attention for image inpainting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019.
[12]YI Z,TANG Q,AZIZI S,et al.Contextual residual aggregation for ultra high-resolution image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020.
[13]NAZERI K,NG E,JOSEPH T,et al.Edgeconnect:Structureguided image inpainting using edge prediction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.2019.
[14]YU J,LIN Z,YANG J,et al.Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019.
[15]YU J,LIN Z,YANG J,et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[16]IIZUKA S,SIMO-SERRA E,ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics(ToG),2017,36(4):1-14.
[17]REN Y,YU X,ZHANG R,et al.Structureflow:Image inpain-ting via structure-aware appearance flow[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019.
[18]WANG Y,CHEN Y C,TAO X,et al.Vcnet:A robust approach to blind image inpainting[C]//Proceedings of the European Conference on Computer Vision.2020.
[19]ZENG Y,LIN Z,LU H,et al.Cr-fill:Generative image inpain-ting with auxiliary contextual reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021.
[20]ZENG Y,LIN Z,YANG J,et al.High-resolution image inpain-ting with iterative confidence feedback and guided upsampling[C]//Proceedings of the European Conference on Computer Vision.2020.
[21]GUO Z,CHEN Z,YU T,et al.Progressive image inpaintingwith full-resolution residual network[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019.
[22]LI J,HE F,ZHANG L,et al.Progressive reconstruction of vi-sual structure for image inpainting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019.
[23]LI J,WANG N,ZHANG L,et al.Recurrent feature reasoning for image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020.
[24]ZHANG H,HU Z,LUO C,et al.Semantic image inpaintingwith progressive generative networks[C]//Proceedings of the 26th ACM International Conference on Multimedia.2018.
[25]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention.2015.
[26]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017.
[27]RADFORD A,METZ L,CHINTALA S.Unsupervised repre-sentation learning with deep convolutional generative adversarial networks[J].arXiv:151106434,2015,
[28]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Ge-nerative adversarial networks[J].Communications of the ACM,2020,63(11):139-144.
[29]LIU H,JIANG B,SONG Y,et al.Rethinking image inpainting via a mutual encoder-decoder with feature equalizations[C]//Proceedings of the European Conference on Computer Vision.2020.
[30]SHARAN L,ROSENHOLTZ R,ADELSON E H.Accuracy and speed of material categorization in real-world images[J].Journal of Vision,2014,14(9):12.
[31]FUKUSHIMA K,MIYAKE S.Neocognitron:A self-organizing neural network model for a mechanism of visual pattern recognition[M].Competition and Cooperation in Neural Nets.Sprin-ger.1982:267-85.
[32]XUE A.End-to-end chinese landscape painting creation using generative adversarial networks[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2021.
[33]HORE A,ZIOU D.Image quality metrics:PSNR vs. SSIM[C]//Proceedings of the 2010 20th International Conference on Pattern Recognition.2010.
[34]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发