计算机科学 ›› 2023, Vol. 50 ›› Issue (1): 147-155.doi: 10.11896/jsjkx.211100161

• 计算机图形学&多媒体 • 上一篇    下一篇

残差注意力与多特征融合的图像去模糊

赵倩, 周冬明, 杨浩, 王长城   

  1. 云南大学信息学院 昆明 650504
  • 收稿日期:2021-11-15 修回日期:2022-04-28 出版日期:2023-01-15 发布日期:2023-01-09
  • 通讯作者: 周冬明(zhoudm@ynu.edu.cn)
  • 作者简介:qian.zhao@mail.ynu.edu.cn
  • 基金资助:
    国家自然科学基金(61966037,62066047)

Image Deblurring Based on Residual Attention and Multi-feature Fusion

ZHAO Qian, ZHOU Dongming, YANG Hao, WANG Changchen   

  1. School of Information Science & Engineering,Yunnan University,Kunming 650504,China
  • Received:2021-11-15 Revised:2022-04-28 Online:2023-01-15 Published:2023-01-09
  • About author:ZHAO Qian,born in 1997,postgra-duate.Her main research interests include deep learning and image restoration.
    ZHOU Dongming,born in 1963,Ph.D,professor.His main research interests include image processing based on deep learning and biological information processing based on machine learning.
  • Supported by:
    National Natural Science Foundation of China(61966037,62066047).

摘要: 动态场景下的非均匀盲去模糊是一个极具挑战性的计算机视觉问题。虽然基于深度学习的去模糊算法已经取得很大进展,但仍存在去模糊不彻底和细节丢失等问题。针对这些问题,提出了一种基于残差注意力和多特征融合的去模糊网络。与现有的单分支网络结构不同,所提网络由两个独立的特征提取子网组成。主干网络采用基于U-Net结构的编码器-解码器网络来获取不同层级的图像特征,并使用残差注意力模块对特征进行筛选,从而自适应地学习图像的轮廓特征和空间结构特征。另外,为了补偿主干网络中下采样操作和上采样操作造成的信息损失,进一步利用具有大感受野的深层次加权残差密集子网来提取特征图的细节信息。最后,使用多特征融合模块逐步融合原分辨率模糊图像以及主干网络和加权残差密集子网生成的特征信息,使得网络能够以整体的方式自适应地学习更有效的特征来复原模糊图像。为了评估网络的去模糊效果,在基准数据集GoPro数据集和HIDE数据集上进行了测试,结果表明所提方法能够有效复原模糊图像。与现有方法相比,提出的去模糊算法在视觉效果上和客观评价指标上均取得了很好的去模糊效果。

关键词: 图像去模糊, 注意力机制, 编码-解码结构, 密集残差网络, 特征融合

Abstract: Non-uniform blind deblurring in dynamic scenes is a challenging computer vision problem.Although deblurring algorithms based on deep learning have made great progress,there are still problems such as incomplete deblurring and loss of details.To solve these problems,a deblurring network based on residual attention and multi-feature fusion is proposed.Unlike the existing single-branch network structure,the proposed network consists of two independent feature extraction subnets.The backbone network uses an encoder-decoder network based on U-Net to obtain image features at different scales,and uses the residual attention module to filter the features,so as to adaptively learn the contour features and spatial structure features of the image.In addition,in order to compensate for the information loss caused by the down-sampling operation and up-sampling operation in the backbone network,a deep weighted residual dense subnet with a large receptive field is further used to extract rich detailed information of the feature map.Finally,the multi-feature fusion module is used to gradually fuse the original resolution blurred image and the feature information generated by the backbone network and the weighted residual dense subnet,so that the network can adaptively learn more effective features in an overall manner to restore the blurred image.In order to evaluate the deblurring performance of the network,tests are conducted on the benchmark data sets GoPro and HIDE,and the results show that the blurred image can be effectively restored.Compared with the existing methods,the proposed deblurring algorithm has achieved excellent deblurring performances in terms of visual effects and objective evaluation indicators.

Key words: Image deblurring, Attention mechanism, Encoding-Decoding structure, Dense residual network, Feature fusion

中图分类号: 

  • TP391
[1]WANG J,SONG L,LI Z,et al.End-to-end object detection with fully convolutional network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:15849-15858.
[2]LI X,HE H,LI X,et al.PointFlow:Flowing semantics through points for aerial image segmentation[C]//Proceedings of the IEEE /CVF Conference on Computer Vision and Pattern Recognition.2021:4217-4226.
[3]DAI P,WENG R,CHOI W,et al.Learning a proposal classifier for multiple object tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:2443-2452.
[4]PAN J S.Research progress on deep learning-based image deblurring[J].Computer Science,2021,48(3):9-13.
[5]KRISHNAN D,TAY T,FERGUS R.Blind deconvolution using a normalized sparsity measure[C]//CVPR 2011.IEEE,2011:233-240.
[6]XU L,ZHENG S,JIA J.Unnatural l0 sparse representation for natural image deblurring[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2013:1107-1114.
[7]PAN J,SUN D,PFISTER H,et al.Blind image deblurring using dark channel prior[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1628-1636.
[8]NAH S,HYUN KIM T,MU LEE K.Deep multi-scale convolu-tional neural network for dynamic scene deblurring[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3883-3891.
[9]TAO X,GAO H,SHEN X,et al.Scale-recurrent network fordeep image deblurring[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2018:8174-8182.
[10]KUPYN O,BUDZAN V,MYKHAILYCH M,et al.Deblurgan:Blind motion deblurring using conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:8183-8192.
[11]WANG J M,LI X F,YE L,et al.Medical Image Deblur Using Generative Adversarial Networks with Channel Attention[J].Computer Science,2021,48(6A):101-106.
[12]MIAO S,ZHU Y X.Differentiable Neural Architecture Search Method for Blind Image Deblurring[J].Computer Engineering,2021,47(9):313-320.
[13]MATSUSHITA Y,OFEK E,GE W,et al.Full-frame video stabilization with motion inpainting[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2006,28(7):1150-1163.
[14]CHO S,WANG J,LEE S.Video deblurring for hand-held cameras using patch-based synthesis[J].ACM Transactions on Graphics(TOG),2012,31(4):1-9.
[15]HYUN KIM T,MU LEE K.Generalized video deblurring for dynamic scenes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:5426-5434.
[16]PAN J,BAI H,TANG J.Cascaded deep video deblurring using temporal sharpness prior[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:3043-3051.
[17]FERGUS R,SINGH B,HERTZMANN A,et al.Removing ca-mera shake from a single photograph[M]//ACM SIGGRAPH 2006 Papers.2006:787-794.
[18]SHAN Q,JIA J,AGARWALA A.High-quality motion deblurring from a single image[J].ACM Transactions on Graphics(tog),2008,27(3):1-10.
[19]HYUN KIM T,AHN B,MU LEE K.Dynamic scene deblurring[C]//Proceedings of the IEEE International Conference on Computer Vision.2013:3160-3167.
[20]HYUN KIM T,MU LEE K.Segmentation-free dynamic scene deblurring[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:2766-2773.
[21]BAI Y,CHEUNG G,LIU X,et al.Graph-based blind image deblurring from a single photograph[J].IEEE Transactions on Image Processing,2018,28(3):1404-1418.
[22]XU L,REN J S,LIU C,et al.Deep convolutional neural network for image deconvolution[J].Advances in Neural Information Processing Systems,2014,27:1790-1798.
[23]SUN J,CAO W,XU Z,et al.Learning a convolutional neuralnetwork for non-uniform motion blur removal[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:769-777.
[24]SCHULER C J,HIRSCH M,HARMELING S,et al.Learning to deblur[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,38(7):1439-1451.
[25]KUPYN O,MARTYNIUK T,WU J,et al.Deblurgan-v2:Deblurring(orders-of-magnitude) faster and better[C]//Procee-dings of the IEEE/CVF International Conference on Computer Vision.2019:8878-8887.
[26]LIU X,SUGANUMA M,SUN Z,et al.Dual residual networks leveraging the potential of paired operations for image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:7007-7016.
[27]ZHANG H,DAI Y,LI H,et al.Deep stacked hierarchical multi-patch network for image deblurring[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:5978-5986.
[28]PARK D,KANG D U,KIM J,et al.Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training[C]//European Conference on Computer Vision.Cham:Springer,2020:327-343.
[29]ZHANG K,LUO W,ZHONG Y,et al.Deblurring by realistic blurring[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:2737-2746.
[30]CHEN L,LU X,ZHANG J,et al.HINet:Half instance normalization network for image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:182-192.
[31]TSAI F J,PENG Y T,LIN Y Y,et al.BANet:Blur-aware attention networks for dynamic scene deblurring[J].arXiv:2101.07518,2021.
[32]LI F,CONG R,BAI H,et al.Learning deep interleaved networks with asymmetric co-attention for image restoration[J].arXiv:2010.15689,2020.
[33]HUYNH-THU Q,GHANBARI M.Scope of validity of PSNR in image/video quality assessment[J].Electronics Letters,2008,44(13):800-801.
[34]SHEN Z,WANG W,LU X,et al.Human-aware motion deblurring[C]//Proceedings of the IEEE/CVF International Confe-rence on Computer Vision.2019:5572-5581.
[35]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[1] 蔡肖, 陈志华, 盛斌.
基于移位窗口金字塔Transformer的遥感图像目标检测
SPT:Swin Pyramid Transformer for Object Detection of Remote Sensing
计算机科学, 2023, 50(1): 105-113. https://doi.org/10.11896/jsjkx.211100208
[2] 张婧媛, 王宏霞, 何沛松.
基于Transformer的多任务图像拼接篡改检测算法
Multitask Transformer-based Network for Image Splicing Manipulation Detection
计算机科学, 2023, 50(1): 114-122. https://doi.org/10.11896/jsjkx.211100269
[3] 李雪辉, 张拥军, 史殿习, 徐化池, 史燕燕.
融合注意力特征的无锚框视觉目标跟踪方法
AFTM:Anchor-free Object Tracking Method with Attention Features
计算机科学, 2023, 50(1): 138-146. https://doi.org/10.11896/jsjkx.211000083
[4] 郑诚, 梅亮, 赵伊研, 张苏航.
基于双向注意力机制和门控图卷积网络的文本分类方法
Text Classification Method Based on Bidirectional Attention and Gated Graph Convolutional Networks
计算机科学, 2023, 50(1): 221-228. https://doi.org/10.11896/jsjkx.211100095
[5] 周芳泉, 成卫青.
基于全局增强图神经网络的序列推荐
Sequence Recommendation Based on Global Enhanced Graph Neural Network
计算机科学, 2022, 49(9): 55-63. https://doi.org/10.11896/jsjkx.210700085
[6] 戴禹, 许林峰.
基于文本行匹配的跨图文本阅读方法
Cross-image Text Reading Method Based on Text Line Matching
计算机科学, 2022, 49(9): 139-145. https://doi.org/10.11896/jsjkx.220600032
[7] 周乐员, 张剑华, 袁甜甜, 陈胜勇.
多层注意力机制融合的序列到序列中国连续手语识别和翻译
Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion
计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[8] 熊丽琴, 曹雷, 赖俊, 陈希亮.
基于值分解的多智能体深度强化学习综述
Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization
计算机科学, 2022, 49(9): 172-182. https://doi.org/10.11896/jsjkx.210800112
[9] 饶志双, 贾真, 张凡, 李天瑞.
基于Key-Value关联记忆网络的知识图谱问答方法
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[10] 汪鸣, 彭舰, 黄飞虎.
基于多时间尺度时空图网络的交通流量预测模型
Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction
计算机科学, 2022, 49(8): 40-48. https://doi.org/10.11896/jsjkx.220100188
[11] 朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥.
基于注意力机制的医学影像深度哈希检索算法
Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism
计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[12] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[13] 闫佳丹, 贾彩燕.
基于双图神经网络信息融合的文本分类方法
Text Classification Method Based on Information Fusion of Dual-graph Neural Network
计算机科学, 2022, 49(8): 230-236. https://doi.org/10.11896/jsjkx.210600042
[14] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
基于改进位置编码的谣言检测模型
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[15] 张颖涛, 张杰, 张睿, 张文强.
全局信息引导的真实图像风格迁移
Photorealistic Style Transfer Guided by Global Information
计算机科学, 2022, 49(7): 100-105. https://doi.org/10.11896/jsjkx.210600036
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!