计算机科学 ›› 2021, Vol. 48 ›› Issue (7): 184-189.doi: 10.11896/jsjkx.200800224

• 计算机图形学&多媒体 • 上一篇    下一篇

基于深度学习特征匹配的视频超分辨率方法

程松盛, 潘金山   

  1. 南京理工大学计算机科学与工程学院 南京210094
  • 收稿日期:2020-08-31 修回日期:2020-10-26 出版日期:2021-07-15 发布日期:2021-07-02
  • 通讯作者: 潘金山(jspan@njust.edu.cn)
  • 基金资助:
    国家自然科学基金(61872421)

Video Super-resolution Method Based on Deep Learning Feature Warping

CHENG Song-sheng, PAN Jin-shan   

  1. School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China
  • Received:2020-08-31 Revised:2020-10-26 Online:2021-07-15 Published:2021-07-02
  • About author:CHENG Song-sheng,postgraduate.His research interests include image super resolution,image/video enhancement and related vision problems.(835357340@qq.com)
    PAN Jin-shan,professor.His main research interests include image deblurring,image/video analysis and enhancement,and related vision problems.
  • Supported by:
    National Natural Science Foundation of China(61872421).

摘要: 视频复原的目标是从给定的退化视频序列中把潜在的高质量视频复原出来。现有的视频复原方法主要集中在如何有效地找到相邻帧之间的运动信息,然后利用运动信息建立相邻帧之间的匹配。与这些方法不同,文中提出了基于深度学习特征匹配的方法来解决视频超分辨率问题。首先,通过深度卷积神经网络计算出相邻帧之间的运动信息;然后,采用一个浅层深度卷积神经网络从输入的视频帧中提取特征,基于估计到的运动信息,将浅层深度卷积神经网络提取到的特征匹配到中间视频帧对应的特征中,并将得到的特征进行有效融合;最后,采用一个深度卷积神经网络重建视频帧。大量的实验结果验证了基于深度学习特征匹配的方法能有效地解决视频超分辨率问题。与现有的基于视频帧匹配的方法相比,所提方法在现有的公开视频超分辨率数据集上取得了较好的效果。

关键词: 深度卷积神经网络, 视频超分辨率, 视频复原, 特征匹配, 运动估计

Abstract: Video restoration aims to restore potential clear videos from a given degraded video sequence.Existing video restoration methods usually focus on modeling the motion information among adjacent frames and establishing the alignment among them.Different from these methods,this paper proposes a feature warping method based on deep learning for video super-resolution.Firstly,the proposed algorithm estimates the motion information based on deep convolutional neural networks.Then,it develops a shallow deep convolutional neural network to estimate the features from input frames.Based on the estimated motion information,this paper warps the deep features to those of the central frames.Next,the proposed method fuses the deep features effectively.Finally,this paper proposes a restoration network which is able to reconstruct clear frames.Experimental results de-monstrate the effectiveness of the proposed algorithm.The proposed algorithm performs well on the benchmark datasets compared to existing methods.

Key words: Deep convolutional neural network, Feature warping, Motion estimation, Video restoration, Video super-resolution

中图分类号: 

  • TP751
[1]FARSIU S,ROBINSON M D,ELAD M,et al.Fast and robust multiframe super resolution[J].IEEE Transactions on Image Processing,2004,13(10):1327-1344.
[2]BASCLE B,BLAKE A,ZISSERMAN A.Motion deblurring and super-resolution from an image sequence[C]//Proceedings of the European Conference on Computer Vision.1996:571-582.
[3]CAIN S C,HAYAT M M,ARMSTRONG E E.Projection-based image registration in the presence of fixed-pattern noise[J].IEEE Transactions on Image Processing,2001,10(12):1860-1872.
[4]SHAHAR O,FAKTOR A,IRANI M.Space-time super-resolution from a single video[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2011:3353-3360.
[5]LIU C,SUN D.On Bayesian adaptive video super resolution[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2014,36(2):346-360.
[6]MA Z,LIAO R,TAO X,et al.Handling motion blur in multi-frame super-resolution[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2015:5224-5232.
[7]DONG C,LOY C C,HE K,et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,38(2):295-307.
[8]KIM J,KWON LEE J,MU LEE K.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1646-1654.
[9]DONG C,LOY C C,TANG X.Accelerating the super-resolution convolutional neural network[C]// Proceedings of the European Conference on Computer Vision.2016:391-407.
[10]HARIS M,SHAKHNAROVICH G,UKITA N.Deep back-projection networks for super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:1664-1673.
[11]ZHANG Y,LI K P,LI K,et al.Image super-resolution usingvery deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision.2018:286-301.
[12]LIM B,SON S,KIM H,et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2017:136-144.
[13]LEDIG C,THEIS L,HUSZAR F,et al.Photo-realistic singleimage super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4681-4690.
[14]SHI W,CABALLERO J,HUSZAR F,et al.Real-time single ima-ge and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1874-1883.
[15]TIAN Y,ZHANG Y,FU Y,et al.TDAN:Temporally-Defor-mable Alignment Network for Video Super-Resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2020:3360-3369.
[16]WANG X,CHAN K C K,YU K,et al.Edvr:Video restoration with enhanced deformable convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:1954-1963.
[17]LIU D,WANG Z,FAN Y,et al.Robust video super-resolution with learned temporal dynamics[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:2507-2515.
[18]CABALLERO J,LEDIG C,AITKEN A,et al.Real-time video super-resolution with spatio-temporal networks and motion compensation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4778-4787.
[19]XUE T,CHEN B,WU J,et al.Video enhancement with task-oriented flow[J].International Journal of Computer Vision,2019,127(8):1106-1125.
[20]JO Y,OH S W,KANG J,et al.Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:3224-3232.
[21]HUANG Y,WANG W,WANG L.Bidirectional recurrent con-volutional networks for multi-frame super-resolution[C]//Advances in Neural Information Processing Systems.2015:235-243.
[22]LIAO R,TAO X,LI R,et al.Video super-resolution via deep draft-ensemble learning [C]//Proceedings of the IEEE International Conference on Computer Vision.2015:531-539.
[23]KAPPELER A,YOO S,DAI Q,et al.Video super-resolution with convolutional neural networks[J].IEEE Transactions on Computational Imaging,2016,2(2):109-122.
[24]TAO X,GAO H,LIAO R,et al.Detail-revealing deep video super-resolution[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:4472-4480.
[25]HARIS M,SHAKHNAROVICH G,UKITA N.Recurrentback-projection network for video super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:3897-3906.
[26]WANG X,YU K,WU S,et al.Esrgan:Enhanced super-resolution generative adversarial networks [C]//Proceedings of the European Conference on Computer Vision Workshops.2018:63-79.
[27]BULAT A,YANG J,TZIMIROPOULOS G.To learn image super-resolution,use a gan to learn how to do image degradation first[C]//Proceedings of the European Conference on Computer Vision.2018:185-200.
[28]CHU M,XIE Y,LEAL-TAIXÉ L,et al.Temporally coherentgans for video super-resolution (tecogan)[J].arXiv:1811.09393,2018,1(2):3.
[29]LUCAS A,LOPEZ-TAPIA S,MOLINA R,et al.Generative adversarial networks and perceptual losses for video super-resolution[J].IEEE Transactions on Image Processing,2019,28(7):3312-3327.
[30]SUN D,YANG X,LIU M Y,et al.Pwc-net:Cnns for optical flow using pyramid,warping,and cost volume[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:8934-8943.
[31]NAH S,BAIK S,HONG S,et al.Ntire 2019 challenge on video deblurring and super-resolution:Dataset and study[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:1996-2005.
[32]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[1] 胡安祥, 尹小康, 朱肖雅, 刘胜利.
基于数据流特征的比较类函数识别方法
Strcmp-like Function Identification Method Based on Data Flow Feature Matching
计算机科学, 2022, 49(9): 326-332. https://doi.org/10.11896/jsjkx.220200163
[2] 孙洁琪, 李亚峰, 张文博, 刘鹏辉.
基于离散小波变换的双域特征融合深度卷积神经网络
Dual-field Feature Fusion Deep Convolutional Neural Network Based on Discrete Wavelet Transformation
计算机科学, 2022, 49(6A): 434-440. https://doi.org/10.11896/jsjkx.210900199
[3] 胡伏原, 万新军, 沈鸣飞, 徐江浪, 姚睿, 陶重犇.
深度卷积神经网络图像实例分割方法研究进展
Survey Progress on Image Instance Segmentation Methods of Deep Convolutional Neural Network
计算机科学, 2022, 49(5): 10-24. https://doi.org/10.11896/jsjkx.210200038
[4] 冷佳旭, 王佳, 莫梦竟成, 陈泰岳, 高新波.
基于深度学习的视频超分辨率重构进展综述
Survey on Video Super-resolution Based on Deep Learning
计算机科学, 2022, 49(2): 123-133. https://doi.org/10.11896/jsjkx.211000007
[5] 刘帅, 芮挺, 胡育成, 杨成松, 王东.
基于深度学习SuperGlue算法的单目视觉里程计
Monocular Visual Odometer Based on Deep Learning SuperGlue Algorithm
计算机科学, 2021, 48(8): 157-161. https://doi.org/10.11896/jsjkx.200700134
[6] 余晗青, 杨贞, 殷志坚.
基于区域激活策略的Tiny YOLOv3目标检测算法
Tiny YOLOv3 Target Detection Algorithm Based on Region Activation Strategy
计算机科学, 2021, 48(6A): 118-121. https://doi.org/10.11896/jsjkx.200700122
[7] 石先让, 宋廷伦, 唐得志, 戴振泳.
一种新颖的单目视觉深度学习算法:H_SFPN
Novel Deep Learning Algorithm for Monocular Vision:H_SFPN
计算机科学, 2021, 48(4): 130-137. https://doi.org/10.11896/jsjkx.200400090
[8] 许虞俊, 李晨.
基于YOLO优化的轻量级目标检测网络
Light-weight Object Detection Network Optimized Based on YOLO Family
计算机科学, 2021, 48(11A): 265-269. https://doi.org/10.11896/jsjkx.201000152
[9] 许华杰, 杨洋, 李桂兰.
基于注意力机制和深度卷积神经网络的材质识别方法
Material Recognition Method Based on Attention Mechanism and Deep Convolutional Neural Network
计算机科学, 2021, 48(10): 220-225. https://doi.org/10.11896/jsjkx.200800073
[10] 高玉潼, 雷为民, 原玥.
复杂环境下基于聚类分析的人脸目标识别
Face Recognition Based on Cluster Analysis in Complex Environment
计算机科学, 2020, 47(7): 111-117. https://doi.org/10.11896/jsjkx.190500004
[11] 蔡于涵,熊淑华,孙伟恒,Karn Pradeep,何小海.
基于运动矢量细化的帧率上变换与HEVC结合的视频压缩算法
Video Compression Algorithm Combining Frame Rate Up-conversion with HEVC Standard Based on Motion Vector Refinement
计算机科学, 2020, 47(2): 76-82. https://doi.org/10.11896/jsjkx.190500092
[12] 李月峰.
一种基于多特征结合的三维模型检索方法
3D Retrieval Algorithm Based on Multi-feature
计算机科学, 2019, 46(6A): 266-269.
[13] 曲佳, 时增林, 叶阳东.
基于卷积特征的非平衡人群密度估计方法
Unbalanced Crowd Density Estimation Based on Convolutional Features
计算机科学, 2018, 45(8): 236-241. https://doi.org/10.11896/j.issn.1002-137X.2018.08.042
[14] 雷倩,郝存明,张伟平.
基于超分辨率和深度神经网络的车型识别
Vehicle Recognition Based on Super-resolution and Deep Neural Networks
计算机科学, 2018, 45(6A): 230-233.
[15] 刘朝霞,邵峰,景雨,祁瑞华.
基于视觉约束能量最小化的特征点匹配算法
Feature Matching Algorithm Based on Visual Feature Constrained Energy Minimization
计算机科学, 2018, 45(5): 228-231. https://doi.org/10.11896/j.issn.1002-137X.2018.05.039
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!