基于残差的端对端图像超分辨率

doi:10.11896/j.issn.1002-137X.2019.06.037

摘要/Abstract

摘要： 深度卷积神经网络使图像超分辨率在准确性方面得到了很大改善。针对基于卷积神经网络的超分辨率重建方法网络结构简单、收敛速度慢、重建纹理模糊等问题,提出了一种基于残差学习的端对端深层卷积神经网络。该网络由局部残差网络和全局残差网络联合训练得到,增加了网络的宽度,能学习到不同的有效特征。局部残差网络包括特征提取、上采样和多尺度重建3个阶段,通过残差密集块密集连接卷积层提取有效的局部特征,采用多尺度卷积层获得丰富的上下文信息,利于高频信息的恢复;全局残差网络中采用渐进上采样的方式实现不同尺度的图像重建,通过残差学习提高收敛速度。在基准数据集Set5,Set14,B100和Urban100上进行放大2倍、3倍和4倍的定量和定性评估。在这4种数据集下,所提算法在放大3倍时平均PSNR/SSIM指标分别为34.70dB/0.9295,30.54dB/0.8490,29.27dB/0.8096和28.81dB/0.8653,与其他方法相比有较大提升。在定性比较方面,所提方法重建出了更加清晰的图像,能更好地保留图像中的边缘细节。实验结果表明,所提方法在主观视觉和客观量化方面都有了较大改进,能有效提高图像重建的质量。

关键词: 残差学习, 超分辨率, 端对端, 卷积神经网络, 联合训练

Abstract: Image super-resolution reconstruction technology is widely used in real life.An end-to-end deep convolutional neural network (CNN) based on residual learning wasproposed to solve the problems of simple network structure,slow convergence rate and reconstructed texture blur in the network super-resolution CNN to further improve the quality of image reconstruction.The network is jointly trained by the local residual network and the global residual network,which increases the width of the network and learns different effective features.The local residual network includes three stages:feature extraction,upsampling and multi-scale reconstruction.The effective local features are extracted by densely concatenated blocks and the rich context information is obtained by multi-scale reconstruction,which is beneficial to the recovery of high-frequency information.In the global residual network,progressive upsampling is used to achieve multi-scale image reconstruction,and the convergence speed is improved by residual learning.Quantitative and qualitative evaluations are performed on the benchmark datasets Set5,Set14,B100,and Urban100 for scale factor of 2,3,and 4.The proposed algorithm shows improved performances by 34.70dB/0.9295,30.54dB/0.8490,29.27dB/0.8096,and 28.81 dB/0.8653 on scale factor of 3.In terms of qualitative comparison,the proposed method reconstructs a clearer image,and preserves the edge details in the image better.The experimental results show that the proposed me-thod has been greatly improved in subjective vision and objective quantization,which can improve the quality of image reconstruction effectively.

Key words: Convolutional neural network, End-to-end, Joint training, Residual learning, Super resolution

中图分类号:

TP391

华臻, 张海程, 李晋江. 基于残差的端对端图像超分辨率[J]. 计算机科学, 2019, 46(6): 246-255. https://doi.org/10.11896/j.issn.1002-137X.2019.06.037

HUA Zhen, ZHANG Hai-cheng, LI Jin-jiang. End-to-end Image Super Resolution Based on Residuals[J]. Computer Science, 2019, 46(6): 246-255. https://doi.org/10.11896/j.issn.1002-137X.2019.06.037

参考文献

[1]BAKER S,KANADE T.Limits on super-resolution and how tobreak them[J].IEEE Transactions on Pattern Analysis andMachine Intelligence,2002,24(9):1167-1183.
[2]XIAO J S,XU Z M,LI S,et al.Interpolation algorithm based on Improved adaptive shock filter image super-resolution[J].Chinese Journal of Computers,2015,38(6):1131-1139.(in Chinese)
肖进胜,饶天宇,贾茜,等.改进的自适应冲击滤波图像超分辨率插值算法[J].计算机学报,2015,38(6):1131-1139.
[3]QIN X J,SHAN Y Y,XIAO J J,et al.Self-learning single image super-resolution reconstruction based on compressive sensing and SVR[J].Computer Science,2017,44(S2):169-174.(in Chinese)
秦绪佳,单扬洋,肖佳吉,等.基于压缩感知和SVR的自学习单幅图像超分辨率重建[J].计算机科学,2017,44(S2):169-174.
[4]HUANG J B,SINGH A,AHUJA N.Single image super-resolution from transformed self-exemplars[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.LosAlamitos:IEEE Computer Society Press,2015:5197-5206.
[5]XIAO J S,GAO W,PENG H,et al.Detail enhancement image super-resolution algorithm based on SVDand local self-similarity[J].Chinese Journal of Computers,2016,39(7):1393-1406.(in Chinese)
肖进胜,高威,彭红,等.基于局部自相似性和奇异值分解的超采样图像细节增强[J].计算机学报,2016,39(7):1393-1406.
[6]FREEMAN W T,PASZTOR E C,CARMICHAEL O T.Lear-ning low-levelvision[J].International Journal of Computer Vision,2000,40(1):25-47 [7]CHANG H,XIONG Y M,YEUNG D Y.Super-resolution through neighbor embedding[C]∥Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2004:275-282 [8]YANG J H,WRIGHT J,HUANG T S,et al.Image super-resolutionvia sparse representation[J].IEEE Transactions on Image Processing,2010,19(11):2861-2873 [9]LI J H,WU Y R,LV J J.Online single image super-resolution algorithm based on group sparse representation[J].Computer Science,2018,45(4):312-318.(in Chinese)
李键红,吴亚榕,吕巨建.基于组稀疏表示的在线单帧图像超分辨率算法[J].计算机科学,2018,45(4):312-318.
[10]YANG C Y,YANG M H.Fast Direct Super-Resolution by Simple Functions[C]∥Proceedings of the IEEE International Conference on Computer Vision.IEEE,2014:561-568.
[11]TIMOFTE R,DE V,VAN GOOL L.Anchored neighborhood regression for fast example-based super-resolution[C]∥Proceedings of the IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2013:1920-1927 [12]TIMOFTE R,SMET V D,GOOL L V.A+:Adjusted Anchored Neighborhood Regression for Fast Super-Resolution[C]∥Proceedings of the Asian Conference on Computer Vision.Cham:Springer,2014:111-126.
[13]DONG C,CHEN C L,HE K,et al.Learning a Deep Convolutional Network for Image Super-Resolution[C]∥Proceedings of the European Conference on Computer Vision.Cham:Springer,2014:184-199.
[14]BENGIO Y,SIMARD P,FRASCONI P.Learning long-term dependencies with gradient descent is difficult[J].IEEE Transactions on Neural Networks,2002,5(2):157-166.
[15]HE K,ZHANG X,REN S,et al.Deep Residual Learning for Ima-ge Recognition[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamites:IEEE Computer Society,2015:770-778.
[16]KIM J,LEE J K,LEE K M.Accurate Image Super-Resolution Using Very Deep Convolutional Networks[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2016:1646-1654.
[17]LAI W S,HUANG J B,AHUJA N,et al.Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamites:IEEE Computer Society,2017:5835-5843.
[18]LIM B,SON S,KIM H,et al.Enhanced Deep Residual Networks for Single Image Super-Resolution[C]∥Proceedings of the Computer Vision and Pattern Recognition Workshops.IEEE,2017:1132-1140.
[19]HARIS M,SHAKHNAROVICH G,UKITA N.Deep Back-Projection Networks For Super-Resolution[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018.
[20]HU Y,GAO X,LI J,et al.Single Image Super-Resolution via Cascaded Multi-Scale Cross Network[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018.
[21]ZAGORUYKO S,KOMODAKIS N.DiracNets:Training Very Deep Neural Networks Without Skip-Connections[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017.
[22]WANG Y,WANG L,WANG H,et al.End-to-End Image Super-Resolution via Deep and Shallow Convolutional Networks[C]∥Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition.IEEE,2016.
[23]REN H,EL-KHAMY M,LEE J.Image Super Resolution Based on Fusing Multiple Convolution Neural Networks[C]∥Proceedings of the Computer Vision & Pattern Recognition.IEEE,2017.
[24]ZHANG Y,TIAN Y,KONG Y,et al.Residual Dense Network for Image Super-Resolution[C]∥Proceedings of the Computer Vision and Pattern Recognition,2018.
[25]KIM J,LEE J K,LEE K M.Deeply-Recursive Convolutional Network for Image Super-Resolution[C]∥Proceedings of the Computer Vision and Pattern Recognition.IEEE,2016:1637-1645.
[26]SHI W,CABALLERO J,HUSZR F,et al.Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2016:1874-1883.
[27]DONG C,CHEN C L,TANG X.Accelerating the Super-Resolution Convolutional Neural Network[C]∥Proceedings of the IEEE International Conference on Computer Vision.IEEE,2016:391-407.
[28]XU B,WANG N Y,CHEN T Q,et al.Empirical evaluationof rectifled activations in convolutional network[C]∥Proceedings of the 32th International Conference on Machine Learning:Deep Learning Workshop.Lille,France:ICML,2015.
[29]LEDIG C,THEIS L,HUSZáR F,et al.Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2017:105-114.
[30]HUANG G,LIU Z,MAATEN L V D,et al.Densely Connected Convolutional Networks[C]∥Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.IEEE Computer Society,2017:2261-2269.
[31]TONG T,LI G,LIU X,et al.Image Super-Resolution Using Dense Skip Connections[C]∥IEEE International Conference on Computer Vision.IEEE Computer Society,2017:4809-4817.
[32]NAIR V,HINTON G E.Rectified linear units improve restric-ted boltzmann machines[C]∥Proceedings of the International Conference on Machine Learning.Omnipress,2010:807-814.
[33]SZEGEDY C,VANHOUCKE V,IOFFE S,et al.Rethinking the Inception Architecture for Computer Vision[C]∥Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition.IEEE,2016:2818-2826.
[34]SZEGEDY C,LIU W,JIA Y,et al.Going Deeper with Convolutions[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2015.
[35]BENGIO Y,SIMARD P,FRASCONI P.Learning long-term dependencies with gradient descent is difficult[J].IEEE Transactions on Neural Networks,2002,5(2):157-166.
[36]BRUHN A,WEICKERT J,SCHN,et al.Lucas/Kanade meets Horn/Schunck:combining local and global optic flow methods[J].International Journal of Computer Vision,2005,61(3):211-231.
[37]PASCANU R,MIKOLOV T,BENGIO Y.On the difficulty of training recurrent neural networks[C]∥Proceedings of the International Conference on International Conference on Machine Learning.JMLR.org,2013:1310-1318.
[38]TIMOFTE R,AGUSTSSON E,GOOL L V,et al.NTIRE 2017 Challenge on Single Image Super-Resolution:Methods and Results[C]∥Proceedings of the Computer Vision and Pattern Re-cognition Workshops.IEEE,2017:1110-1121.
[39]BEVILACQUA M,ROUMY A,GUILLEMOT C,et al.Low-Complexity Single Image Super-Resolution Based on Nonnegative Neighbor Embedding[C]∥Proceedings of the British Machine Vision Conference.BMVA Press,2012:135.1-135.10.
[40]ZEYDE R,ELAD M,PROTTER M.On single image scale-up using sparse-representations[C]∥Proceedings of the International Conference on Curves and Surfaces.Berlin Heidelberg:Springer,2010:711-730.
[41]MARTIN D R,FOWLKES C,TAL D,et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]∥Proceedings of the IEEE International Conference on Computer Vision.IEEE,2002:1110-1121.
[42]HUANG J B,SINGH A,AHUJA N.Single image super-resolution from transformed self-exemplars[C]∥Proceedings of theComputer Vision and Pattern Recognition.IEEE,2015:5197-5206.
[43]JIA Y Q,SHELHAMER,et al.Caffe:Convolutional Architec-ture for Fast Feature Embedding[C]∥Proceedings of the ACM International Conference Multimedia.ACM,2014:675-678.
[44]HE K,ZHANG X,REN S,et al.Delving Deep into Rectifiers:Surpassing Human-Level Performance on ImageNet Classification[C]∥Proceedings of the IEEE International Conference on Computer Vision.IEEE Computer Society,2015:1026-1034.

相关文章 15

[1]	周乐员, 张剑华, 袁甜甜, 陈胜勇. 多层注意力机制融合的序列到序列中国连续手语识别和翻译 Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion 计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[2]	李宗民, 张玉鹏, 刘玉杰, 李华. 基于可变形图卷积的点云表征学习 Deformable Graph Convolutional Networks Based Point Cloud Representation Learning 计算机科学, 2022, 49(8): 273-278. https://doi.org/10.11896/jsjkx.210900023
[3]	陈泳全, 姜瑛. 基于卷积神经网络的APP用户行为分析方法 Analysis Method of APP User Behavior Based on Convolutional Neural Network 计算机科学, 2022, 49(8): 78-85. https://doi.org/10.11896/jsjkx.210700121
[4]	朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥. 基于注意力机制的医学影像深度哈希检索算法 Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism 计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[5]	檀莹莹, 王俊丽, 张超波. 基于图卷积神经网络的文本分类方法研究综述 Review of Text Classification Methods Based on Graph Convolutional Network 计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064
[6]	金方焱, 王秀利. 融合RACNN和BiLSTM的金融领域事件隐式因果关系抽取 Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM 计算机科学, 2022, 49(7): 179-186. https://doi.org/10.11896/jsjkx.210500190
[7]	张颖涛, 张杰, 张睿, 张文强. 全局信息引导的真实图像风格迁移 Photorealistic Style Transfer Guided by Global Information 计算机科学, 2022, 49(7): 100-105. https://doi.org/10.11896/jsjkx.210600036
[8]	戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮. 基于DNGAN的磁共振图像超分辨率重建算法 Super-resolution Reconstruction of MRI Based on DNGAN 计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105
[9]	刘月红, 牛少华, 神显豪. 基于卷积神经网络的虚拟现实视频帧内预测编码 Virtual Reality Video Intraframe Prediction Coding Based on Convolutional Neural Network 计算机科学, 2022, 49(7): 127-131. https://doi.org/10.11896/jsjkx.211100179
[10]	徐鸣珂, 张帆. Head Fusion:一种提高语音情绪识别的准确性和鲁棒性的方法 Head Fusion:A Method to Improve Accuracy and Robustness of Speech Emotion Recognition 计算机科学, 2022, 49(7): 132-141. https://doi.org/10.11896/jsjkx.210100085
[11]	杨玥, 冯涛, 梁虹, 杨扬. 融合交叉注意力机制的图像任意风格迁移 Image Arbitrary Style Transfer via Criss-cross Attention 计算机科学, 2022, 49(6A): 345-352. https://doi.org/10.11896/jsjkx.210700236
[12]	杨健楠, 张帆. 一种结合双注意力机制和层次网络结构的细碎农作物分类方法 Classification Method for Small Crops Combining Dual Attention Mechanisms and Hierarchical Network Structure 计算机科学, 2022, 49(6A): 353-357. https://doi.org/10.11896/jsjkx.210200169
[13]	杨涵, 万游, 蔡洁萱, 方铭宇, 吴卓超, 金扬, 钱伟行. 基于步态分类辅助的虚拟IMU的行人导航方法 Pedestrian Navigation Method Based on Virtual Inertial Measurement Unit Assisted by GaitClassification 计算机科学, 2022, 49(6A): 759-763. https://doi.org/10.11896/jsjkx.211200148
[14]	王杉, 徐楚怡, 师春香, 张瑛. 基于CNN-LSTM的卫星云图云分类方法研究 Study on Cloud Classification Method of Satellite Cloud Images Based on CNN-LSTM 计算机科学, 2022, 49(6A): 675-679. https://doi.org/10.11896/jsjkx.210300177
[15]	孙福权, 崔志清, 邹彭, 张琨. 基于多尺度特征的脑肿瘤分割算法 Brain Tumor Segmentation Algorithm Based on Multi-scale Features 计算机科学, 2022, 49(6A): 12-16. https://doi.org/10.11896/jsjkx.210700217

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed