Computer Science ›› 2023, Vol. 50 ›› Issue (1): 147-155.doi: 10.11896/jsjkx.211100161

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Image Deblurring Based on Residual Attention and Multi-feature Fusion

ZHAO Qian, ZHOU Dongming, YANG Hao, WANG Changchen   

  1. School of Information Science & Engineering,Yunnan University,Kunming 650504,China
  • Received:2021-11-15 Revised:2022-04-28 Online:2023-01-15 Published:2023-01-09
  • About author:ZHAO Qian,born in 1997,postgra-duate.Her main research interests include deep learning and image restoration.
    ZHOU Dongming,born in 1963,Ph.D,professor.His main research interests include image processing based on deep learning and biological information processing based on machine learning.
  • Supported by:
    National Natural Science Foundation of China(61966037,62066047).

Abstract: Non-uniform blind deblurring in dynamic scenes is a challenging computer vision problem.Although deblurring algorithms based on deep learning have made great progress,there are still problems such as incomplete deblurring and loss of details.To solve these problems,a deblurring network based on residual attention and multi-feature fusion is proposed.Unlike the existing single-branch network structure,the proposed network consists of two independent feature extraction subnets.The backbone network uses an encoder-decoder network based on U-Net to obtain image features at different scales,and uses the residual attention module to filter the features,so as to adaptively learn the contour features and spatial structure features of the image.In addition,in order to compensate for the information loss caused by the down-sampling operation and up-sampling operation in the backbone network,a deep weighted residual dense subnet with a large receptive field is further used to extract rich detailed information of the feature map.Finally,the multi-feature fusion module is used to gradually fuse the original resolution blurred image and the feature information generated by the backbone network and the weighted residual dense subnet,so that the network can adaptively learn more effective features in an overall manner to restore the blurred image.In order to evaluate the deblurring performance of the network,tests are conducted on the benchmark data sets GoPro and HIDE,and the results show that the blurred image can be effectively restored.Compared with the existing methods,the proposed deblurring algorithm has achieved excellent deblurring performances in terms of visual effects and objective evaluation indicators.

Key words: Image deblurring, Attention mechanism, Encoding-Decoding structure, Dense residual network, Feature fusion

CLC Number: 

  • TP391
[1]WANG J,SONG L,LI Z,et al.End-to-end object detection with fully convolutional network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:15849-15858.
[2]LI X,HE H,LI X,et al.PointFlow:Flowing semantics through points for aerial image segmentation[C]//Proceedings of the IEEE /CVF Conference on Computer Vision and Pattern Recognition.2021:4217-4226.
[3]DAI P,WENG R,CHOI W,et al.Learning a proposal classifier for multiple object tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:2443-2452.
[4]PAN J S.Research progress on deep learning-based image deblurring[J].Computer Science,2021,48(3):9-13.
[5]KRISHNAN D,TAY T,FERGUS R.Blind deconvolution using a normalized sparsity measure[C]//CVPR 2011.IEEE,2011:233-240.
[6]XU L,ZHENG S,JIA J.Unnatural l0 sparse representation for natural image deblurring[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2013:1107-1114.
[7]PAN J,SUN D,PFISTER H,et al.Blind image deblurring using dark channel prior[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1628-1636.
[8]NAH S,HYUN KIM T,MU LEE K.Deep multi-scale convolu-tional neural network for dynamic scene deblurring[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3883-3891.
[9]TAO X,GAO H,SHEN X,et al.Scale-recurrent network fordeep image deblurring[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2018:8174-8182.
[10]KUPYN O,BUDZAN V,MYKHAILYCH M,et al.Deblurgan:Blind motion deblurring using conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:8183-8192.
[11]WANG J M,LI X F,YE L,et al.Medical Image Deblur Using Generative Adversarial Networks with Channel Attention[J].Computer Science,2021,48(6A):101-106.
[12]MIAO S,ZHU Y X.Differentiable Neural Architecture Search Method for Blind Image Deblurring[J].Computer Engineering,2021,47(9):313-320.
[13]MATSUSHITA Y,OFEK E,GE W,et al.Full-frame video stabilization with motion inpainting[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2006,28(7):1150-1163.
[14]CHO S,WANG J,LEE S.Video deblurring for hand-held cameras using patch-based synthesis[J].ACM Transactions on Graphics(TOG),2012,31(4):1-9.
[15]HYUN KIM T,MU LEE K.Generalized video deblurring for dynamic scenes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:5426-5434.
[16]PAN J,BAI H,TANG J.Cascaded deep video deblurring using temporal sharpness prior[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:3043-3051.
[17]FERGUS R,SINGH B,HERTZMANN A,et al.Removing ca-mera shake from a single photograph[M]//ACM SIGGRAPH 2006 Papers.2006:787-794.
[18]SHAN Q,JIA J,AGARWALA A.High-quality motion deblurring from a single image[J].ACM Transactions on Graphics(tog),2008,27(3):1-10.
[19]HYUN KIM T,AHN B,MU LEE K.Dynamic scene deblurring[C]//Proceedings of the IEEE International Conference on Computer Vision.2013:3160-3167.
[20]HYUN KIM T,MU LEE K.Segmentation-free dynamic scene deblurring[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:2766-2773.
[21]BAI Y,CHEUNG G,LIU X,et al.Graph-based blind image deblurring from a single photograph[J].IEEE Transactions on Image Processing,2018,28(3):1404-1418.
[22]XU L,REN J S,LIU C,et al.Deep convolutional neural network for image deconvolution[J].Advances in Neural Information Processing Systems,2014,27:1790-1798.
[23]SUN J,CAO W,XU Z,et al.Learning a convolutional neuralnetwork for non-uniform motion blur removal[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:769-777.
[24]SCHULER C J,HIRSCH M,HARMELING S,et al.Learning to deblur[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,38(7):1439-1451.
[25]KUPYN O,MARTYNIUK T,WU J,et al.Deblurgan-v2:Deblurring(orders-of-magnitude) faster and better[C]//Procee-dings of the IEEE/CVF International Conference on Computer Vision.2019:8878-8887.
[26]LIU X,SUGANUMA M,SUN Z,et al.Dual residual networks leveraging the potential of paired operations for image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:7007-7016.
[27]ZHANG H,DAI Y,LI H,et al.Deep stacked hierarchical multi-patch network for image deblurring[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:5978-5986.
[28]PARK D,KANG D U,KIM J,et al.Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training[C]//European Conference on Computer Vision.Cham:Springer,2020:327-343.
[29]ZHANG K,LUO W,ZHONG Y,et al.Deblurring by realistic blurring[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:2737-2746.
[30]CHEN L,LU X,ZHANG J,et al.HINet:Half instance normalization network for image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:182-192.
[31]TSAI F J,PENG Y T,LIN Y Y,et al.BANet:Blur-aware attention networks for dynamic scene deblurring[J].arXiv:2101.07518,2021.
[32]LI F,CONG R,BAI H,et al.Learning deep interleaved networks with asymmetric co-attention for image restoration[J].arXiv:2010.15689,2020.
[33]HUYNH-THU Q,GHANBARI M.Scope of validity of PSNR in image/video quality assessment[J].Electronics Letters,2008,44(13):800-801.
[34]SHEN Z,WANG W,LU X,et al.Human-aware motion deblurring[C]//Proceedings of the IEEE/CVF International Confe-rence on Computer Vision.2019:5572-5581.
[35]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[1] CAI Xiao, CEHN Zhihua, SHENG Bin. SPT:Swin Pyramid Transformer for Object Detection of Remote Sensing [J]. Computer Science, 2023, 50(1): 105-113.
[2] ZHANG Jingyuan, WANG Hongxia, HE Peisong. Multitask Transformer-based Network for Image Splicing Manipulation Detection [J]. Computer Science, 2023, 50(1): 114-122.
[3] LI Xuehui, ZHANG Yongjun, SHI Dianxi, XU Huachi, SHI Yanyan. AFTM:Anchor-free Object Tracking Method with Attention Features [J]. Computer Science, 2023, 50(1): 138-146.
[4] ZHENG Cheng, MEI Liang, ZHAO Yiyan, ZHANG Suhang. Text Classification Method Based on Bidirectional Attention and Gated Graph Convolutional Networks [J]. Computer Science, 2023, 50(1): 221-228.
[5] ZHOU Fang-quan, CHENG Wei-qing. Sequence Recommendation Based on Global Enhanced Graph Neural Network [J]. Computer Science, 2022, 49(9): 55-63.
[6] DAI Yu, XU Lin-feng. Cross-image Text Reading Method Based on Text Line Matching [J]. Computer Science, 2022, 49(9): 139-145.
[7] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[8] XIONG Li-qin, CAO Lei, LAI Jun, CHEN Xi-liang. Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization [J]. Computer Science, 2022, 49(9): 172-182.
[9] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[10] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[11] WANG Ming, PENG Jian, HUANG Fei-hu. Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction [J]. Computer Science, 2022, 49(8): 40-48.
[12] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[13] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[14] YAN Jia-dan, JIA Cai-yan. Text Classification Method Based on Information Fusion of Dual-graph Neural Network [J]. Computer Science, 2022, 49(8): 230-236.
[15] ZHANG Ying-tao, ZHANG Jie, ZHANG Rui, ZHANG Wen-qiang. Photorealistic Style Transfer Guided by Global Information [J]. Computer Science, 2022, 49(7): 100-105.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!