多跳连接残差注意网络的图像超分辨率重建

doi:10.11896/jsjkx.201000033

摘要/Abstract

摘要： 随着卷积神经网络深度的不断增加,深度卷积神经网络的训练会变得更加困难。此外,在图像超分辨率中,低分辨率图像的通道特征和输入通常在不同的通道中被平等对待,这就导致了卷积神经网络的表征能力被弱化。为了解决这些问题,提出了一种多跳连接残差注意网络,该网络利用多跳连接中的残差(Residual in Multi-skip Connection,RIMC),构造了具有多个残差组的深度网络。每个残差组包含了一定数量的短跳连接和多跳连接。在RIMC的基础上,主网络被允许穿过多跳连接来绕过丰富的低频信息,同时高频信息也可以被主网络集中地学习。另外,考虑到通道和空间维度的相互依赖关系,提出了注意机制块(Attention Mechanism Block,AMBlock)来关注信息的位置,并自适应地调整通道特征尺度,其中通道注意机制和空间注意机制被应用在这种方式中。实验结果表明,该网络可以更好地恢复图像细节,获得更高的图像质量和网络性能。

关键词: 残差网络, 多跳连接中的残差, 跳连接, 图像超分辨率, 注意机制块

Abstract: Deep convolutional neural networks (Deep CNNs) are difficult to train as they become deeper.Moreover,in image super-resolution,channel-wise features and inputs of the low-resolution (LR) image are treated equally between different channels,resulting in the deficiency of the representational ability of the CNNs.To resolve these issues,residual attention network with multi-skip Connection (RANMC) is proposed for single-image super resolution (SISR),which employs residual in multi-skip connection (RIMC) structure,then a very deep network is formulated with serval residual groups.Each residual group (RG) contains a certain number of short skip connections (SSC) and multi-skip connections (MC).Based on RIMC,rich low-frequency (LF) information is allowed to be bypassed through multi-skip connection,and high-frequency (HF) information is focused on learning by the principal network.Furthermore,considering interdependencies in channel and spatial dimension,attention mechanism block(AMBlock) is proposed to focus on the location of the information and adaptively readjust channel-wise features,where the spatial attention (SA) mechanism and channel attention (CA) mechanism are taken in the approach.Experiments indicate that RANMC can not only recover image details better,but also obtain higher image quality and network performance.

Key words: Attention mechanism block, Image super-resolution, Residual in multi-skip connection, Residual network, Skip connection

中图分类号:

TP391.41

刘遵雄, 朱成佳, 黄稷, 蔡体健. 多跳连接残差注意网络的图像超分辨率重建[J]. 计算机科学, 2021, 48(11): 258-267. https://doi.org/10.11896/jsjkx.201000033

LIU Zun-xiong, ZHU Cheng-jia, HUANG Ji, CAI Ti-jian. Image Super-resolution by Residual Attention Network with Multi-skip Connection[J]. Computer Science, 2021, 48(11): 258-267. https://doi.org/10.11896/jsjkx.201000033

参考文献

[1]IRANI M,PELEG S.Improving resolution by image registration[J].GVGIP:Graphical Models and Image Processing,1991,53(3):231-239.
[2]BEVILACQUA M,ROUMY A,GUILLEMOT C,et al.Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding[C]//BMVC.2012.
[3]DONG C,LOY C C,HE K,et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,38(2):295-307.
[4]WANG Y,LI A,WANG S Q.Research on the influence oftraining set automatic target recognition under super-resolution of remote sensing images[J].Journal of Chongqing University of Technology(Natural Science),2021,35(2):136-143.
[5]WANG W,HU T,LI X W,et al.Study on Super-resolutionImage Reconstruction of Leukocytes[J].Computer Science,2021,48(4):164-168.
[6]ZOU W W W,YUEN P C.Very Low Resolution Face Recognition Problem[J].IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society,2012,21(1):327-340.
[7] KARRAS T,AILA T,LAINE S,et al. Progressive Growing of GANs for Improved Quality,Stability,and Variation[J].arXiv:1710.10196,2018.
[8]CABALLERO J.Cardiac image super-resolution with global correspondence using multi-atlas patchmatch[J].Medical Image Computing and Computer-assisted Intervention,2013,16(Pt3):9-16.
[9]TIMOFTE R,DE V,GOOL L V.Anchored Neighborhood Regression for Fast Example-Based Super-Resolution[C]//IEEE International Conference on Computer Vision.IEEE,2014.
[10]TIMOFTE R,VINCENT D S,LUC V G.A+:Adjusted Anchored Neighborhood Regression for Fast Super-Resolution[C]//Asian Conference on Computer Vision.Cham:Springer,2014.
[11]PELEG T,ELAD M.A Statistical Prediction Model Based on Sparse Representations for Single Image Super-Resolution[J].IEEE Transactions on Image Processing,2014,23(6):2569-2582.
[12]LAI W S,HUANG J B,AHUJA N,et al.Fast and AccurateImage Super-Resolution with Deep Laplacian Pyramid Networks[C]//IEEE Transactions on Pattern Analysis and Machine Intelligence.2017.
[13]HUANG J B,SINGH A,AHUJA N.Single Image Super-resolution from Transformed Self-Exemplars[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2015.
[14]LEE C Y,XIE S,GALLAGHER P,et al.Deeply-supervised nets[C]//Artificial Intelligence and Statistics.2015:562-570.
[15]LI K,WU Z,PENG K C,et al.Tell me where to look:Guided attention inference network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:9215-9223.
[16]ZHANG K,GAO X,TAO D,et al.Single image super-resolution with non-local means and steering kernel regression[J].IEEE Transactions on Image Processing,2012,21(11):4544-4556.
[17]ZHANG L,WU X.An edge-guided image interpolation algo-rithm via directional filtering and data fusion[J].IEEE transactions on Image Processing,2006,15(8):2226-2238.
[18]CHAO D,CHEN C L,HE K,et al.Learning a Deep Convolutional Network for Image Super-Resolution[C]//ECCV.Sprin-ger International Publishing.2014.
[19]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[20]LIM B,SON S,KIM H,et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2017:136-144.
[21]TONG T,LI G,LIU X,et al.Image super-resolution usingdense skip connections[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:4799-4807.
[22]HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[23]WANG F,JIANG M,QIAN C,et al.Residual attention network for image classification[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2017:3156-3164.
[24]KIM J,KWON L J,MU L K.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1646-1654.
[25]KIM J,KWON L J,MU L K.Deeply-recursive convolutionalnetwork for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1637-1645.
[26]DONG C,LOY C C,TANG X.Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision.Cham:Springer,2016:391-407.
[27]NAIR V,HINTON G E.Rectified linear units improve restric-ted boltzmann machines[C]//ICML.2010.
[28]SHI W,CABALLERO J,HUSZÁR F,et al.Real-time singleimage and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1874-1883.
[29]YANG C Y,YANG M H.Fast direct super-resolution by simple functions[C]//Proceedings of the IEEE International Confe-rence on Computer Vision.2013:561-568.
[30]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[31]LIM B,SON S,KIM H,et al.Enhanced Deep Residual Networks for Single Image Super-Resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).IEEE,2017.
[32]CHATTERJEE S,CHU W T.Thermal Face Recognition Based on Transformation by Residual U-Net and Pixel Shuffle Upsampling[M]//Multimedia Modeling,2020:679-689.
[33]TIMOFTE R,AGUSTSSON E,VAN G L,et al.Ntire 2017challenge on single image super-resolution:Methods and results[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2017:114-125.
[34]BEVILACQUA M,ROUMY A,GUILLEMOT C,et al.Low-Complexity Single Image Super-Resolution Based on Nonnegative Neighbor Embedding[C]//BMVC.2012.
[35]ZEYDE R,ELAD M,PROTTER M.On single image scale-up using sparse-representations[C]//International Conference on Curves and Surfaces.Berlin:Springer,2010:711-730.
[36]MARTIN D,FOWLKES C,TAL D,et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings Eighth IEEE International Conference on Computer Vision.2001:416-423.
[37]HUANG J B,SINGH A,AHUJA N.Single image super-resolution from transformed self-exemplars[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:5197-5206.
[38]MATSUI Y,ITO K,ARAMAKI Y,et al.Sketch-based mangaretrieval using manga109 dataset[J].Multimedia Tools and Applications,2017,76(20):21811-21838.
[39]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[40]WANG Z,LIU D,YANG J,et al.Deep networks for image super-resolution with sparse prior[C]//Proceedings of the IEEE International Conference on Computer Vision.2015:370-378.
[41]TAI Y,YANG J,LIU X.Image super-resolution via deep recursive residual network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3147-3155.
[42]ZHANG K,ZUO W,ZHANG L.Learning a single convolutional super-resolution network for multiple degradations[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:3262-3271.
[43]HARIS M,SHAKHNAROVICH G,UKITA N.Deep back-projection networks for super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:1664-1673.
[44]TAI Y,YANG J,LIU X,et al.Memnet:A persistent memory network for image restoration[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:4539-4547.
[45]LAI W S,HUANG J B,AHUJA N,et al.Fast and accurateimage super-resolution with deep laplacian pyramid networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,41(11):2599-2613.
[46]LAI W S,HUANG J B,AHUJA N,et al.Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:624-632.
[47]LIU D,WEN B,FAN Y,et al.Non-local recurrent network for image restoration[C]//Advances in Neural Information Processing Systems.2018:1673-1682.
[48]ZHANG Y,TIAN Y,KONG Y,et al.Residual dense network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:2472-2481.

相关文章 15

[1]	王馨彤, 王璇, 孙知信. 基于多尺度记忆残差网络的网络流量异常检测模型 Network Traffic Anomaly Detection Method Based on Multi-scale Memory Residual Network 计算机科学, 2022, 49(8): 314-322. https://doi.org/10.11896/jsjkx.220200011
[2]	高荣华, 白强, 王荣, 吴华瑞, 孙想. 改进注意力机制的多叉树网络多作物早期病害识别方法 Multi-tree Network Multi-crop Early Disease Recognition Method Based on Improved Attention Mechanism 计算机科学, 2022, 49(6A): 363-369. https://doi.org/10.11896/jsjkx.210500044
[3]	赵人行, 徐频捷, 刘瑶. 基于深度卷积残差网络的心电单导联房颤检测方法 ECG-based Atrial Fibrillation Detection Based on Deep Convolutional Residual Neural Network 计算机科学, 2022, 49(5): 186-193. https://doi.org/10.11896/jsjkx.220200002
[4]	韩红旗, 冉亚鑫, 张运良, 桂婕, 高雄, 易梦琳. 基于共同子空间分类学习的跨媒体检索研究 Study on Cross-media Information Retrieval Based on Common Subspace Classification Learning 计算机科学, 2022, 49(5): 33-42. https://doi.org/10.11896/jsjkx.210200157
[5]	高心悦, 田汉民. 基于改进U-Net网络的液滴分割方法 Droplet Segmentation Method Based on Improved U-Net Network 计算机科学, 2022, 49(4): 227-232. https://doi.org/10.11896/jsjkx.210300193
[6]	张红民, 李萍萍, 房晓冰, 刘宏. 改进YOLOv3网络模型的人体异常行为检测方法 Human Abnormal Behavior Detection Method Based on Improved YOLOv3 Network Model 计算机科学, 2022, 49(4): 233-238. https://doi.org/10.11896/jsjkx.210300251
[7]	瞿中, 陈雯. 基于空洞卷积和多特征融合的混凝土路面裂缝检测 Concrete Pavement Crack Detection Based on Dilated Convolution and Multi-features Fusion 计算机科学, 2022, 49(3): 192-196. https://doi.org/10.11896/jsjkx.210100164
[8]	郭琳, 李晨, 陈晨, 赵睿, 范仕霖, 徐星雨. 基于通道注意递归残差网络的图像超分辨率重建 Image Super-resolution Reconstruction Using Recursive ResidualNetwork Based on ChannelAttention 计算机科学, 2021, 48(8): 139-144. https://doi.org/10.11896/jsjkx.200500150
[9]	许华杰, 张晨强, 苏国韶. 基于深层卷积残差网络的航拍图建筑物精确分割方法 Accurate Segmentation Method of Aerial Photography Buildings Based on Deep Convolutional Residual Network 计算机科学, 2021, 48(8): 169-174. https://doi.org/10.11896/jsjkx.200500096
[10]	暴雨轩, 芦天亮, 杜彦辉, 石达. 基于i_ResNet34模型和数据增强的深度伪造视频检测方法 Deepfake Videos Detection Method Based on i_ResNet34 Model and Data Augmentation 计算机科学, 2021, 48(7): 77-85. https://doi.org/10.11896/jsjkx.210300258
[11]	王建明, 黎向锋, 叶磊, 左敦稳, 张丽萍. 基于信道注意结构的生成对抗网络医学图像去模糊 Medical Image Deblur Using Generative Adversarial Networks with Channel Attention 计算机科学, 2021, 48(6A): 101-106. https://doi.org/10.11896/jsjkx.200600144
[12]	牛康力, 谌雨章, 张龚平, 谭前程, 王绎冲, 罗美琪. 基于深度学习的无人机航拍车流量监测 Vehicle Flow Measuring of UVA Based on Deep Learning 计算机科学, 2021, 48(6A): 275-280. https://doi.org/10.11896/jsjkx.200900149
[13]	龚航, 刘培顺. 夜间行驶车辆远光灯检测方法 Detection Method of High Beam in Night Driving Vehicle 计算机科学, 2021, 48(12): 256-263. https://doi.org/10.11896/jsjkx.200700026
[14]	柴冰, 李冬冬, 王喆, 高大启. 融合频率和通道卷积注意的脑电(EEG)情感识别 EEG Emotion Recognition Based on Frequency and Channel Convolutional Attention 计算机科学, 2021, 48(12): 312-318. https://doi.org/10.11896/jsjkx.201000141
[15]	杨月麟, 毕宗泽. 基于深度学习的网络流量异常检测 Network Anomaly Detection Based on Deep Learning 计算机科学, 2021, 48(11A): 540-546. https://doi.org/10.11896/jsjkx.201200077

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed