基于深度学习特征匹配的视频超分辨率方法

doi:10.11896/jsjkx.200800224

Abstract

Abstract: Video restoration aims to restore potential clear videos from a given degraded video sequence.Existing video restoration methods usually focus on modeling the motion information among adjacent frames and establishing the alignment among them.Different from these methods,this paper proposes a feature warping method based on deep learning for video super-resolution.Firstly,the proposed algorithm estimates the motion information based on deep convolutional neural networks.Then,it develops a shallow deep convolutional neural network to estimate the features from input frames.Based on the estimated motion information,this paper warps the deep features to those of the central frames.Next,the proposed method fuses the deep features effectively.Finally,this paper proposes a restoration network which is able to reconstruct clear frames.Experimental results de-monstrate the effectiveness of the proposed algorithm.The proposed algorithm performs well on the benchmark datasets compared to existing methods.

Key words: Deep convolutional neural network, Feature warping, Motion estimation, Video restoration, Video super-resolution

CLC Number:

TP751

CHENG Song-sheng, PAN Jin-shan. Video Super-resolution Method Based on Deep Learning Feature Warping[J].Computer Science, 2021, 48(7): 184-189.

References

[1]FARSIU S,ROBINSON M D,ELAD M,et al.Fast and robust multiframe super resolution[J].IEEE Transactions on Image Processing,2004,13(10):1327-1344.
[2]BASCLE B,BLAKE A,ZISSERMAN A.Motion deblurring and super-resolution from an image sequence[C]//Proceedings of the European Conference on Computer Vision.1996:571-582.
[3]CAIN S C,HAYAT M M,ARMSTRONG E E.Projection-based image registration in the presence of fixed-pattern noise[J].IEEE Transactions on Image Processing,2001,10(12):1860-1872.
[4]SHAHAR O,FAKTOR A,IRANI M.Space-time super-resolution from a single video[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2011:3353-3360.
[5]LIU C,SUN D.On Bayesian adaptive video super resolution[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2014,36(2):346-360.
[6]MA Z,LIAO R,TAO X,et al.Handling motion blur in multi-frame super-resolution[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2015:5224-5232.
[7]DONG C,LOY C C,HE K,et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,38(2):295-307.
[8]KIM J,KWON LEE J,MU LEE K.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1646-1654.
[9]DONG C,LOY C C,TANG X.Accelerating the super-resolution convolutional neural network[C]// Proceedings of the European Conference on Computer Vision.2016:391-407.
[10]HARIS M,SHAKHNAROVICH G,UKITA N.Deep back-projection networks for super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:1664-1673.
[11]ZHANG Y,LI K P,LI K,et al.Image super-resolution usingvery deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision.2018:286-301.
[12]LIM B,SON S,KIM H,et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2017:136-144.
[13]LEDIG C,THEIS L,HUSZAR F,et al.Photo-realistic singleimage super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4681-4690.
[14]SHI W,CABALLERO J,HUSZAR F,et al.Real-time single ima-ge and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1874-1883.
[15]TIAN Y,ZHANG Y,FU Y,et al.TDAN:Temporally-Defor-mable Alignment Network for Video Super-Resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2020:3360-3369.
[16]WANG X,CHAN K C K,YU K,et al.Edvr:Video restoration with enhanced deformable convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:1954-1963.
[17]LIU D,WANG Z,FAN Y,et al.Robust video super-resolution with learned temporal dynamics[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:2507-2515.
[18]CABALLERO J,LEDIG C,AITKEN A,et al.Real-time video super-resolution with spatio-temporal networks and motion compensation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4778-4787.
[19]XUE T,CHEN B,WU J,et al.Video enhancement with task-oriented flow[J].International Journal of Computer Vision,2019,127(8):1106-1125.
[20]JO Y,OH S W,KANG J,et al.Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:3224-3232.
[21]HUANG Y,WANG W,WANG L.Bidirectional recurrent con-volutional networks for multi-frame super-resolution[C]//Advances in Neural Information Processing Systems.2015:235-243.
[22]LIAO R,TAO X,LI R,et al.Video super-resolution via deep draft-ensemble learning [C]//Proceedings of the IEEE International Conference on Computer Vision.2015:531-539.
[23]KAPPELER A,YOO S,DAI Q,et al.Video super-resolution with convolutional neural networks[J].IEEE Transactions on Computational Imaging,2016,2(2):109-122.
[24]TAO X,GAO H,LIAO R,et al.Detail-revealing deep video super-resolution[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:4472-4480.
[25]HARIS M,SHAKHNAROVICH G,UKITA N.Recurrentback-projection network for video super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:3897-3906.
[26]WANG X,YU K,WU S,et al.Esrgan:Enhanced super-resolution generative adversarial networks [C]//Proceedings of the European Conference on Computer Vision Workshops.2018:63-79.
[27]BULAT A,YANG J,TZIMIROPOULOS G.To learn image super-resolution,use a gan to learn how to do image degradation first[C]//Proceedings of the European Conference on Computer Vision.2018:185-200.
[28]CHU M,XIE Y,LEAL-TAIXÉ L,et al.Temporally coherentgans for video super-resolution (tecogan)[J].arXiv:1811.09393,2018,1(2):3.
[29]LUCAS A,LOPEZ-TAPIA S,MOLINA R,et al.Generative adversarial networks and perceptual losses for video super-resolution[J].IEEE Transactions on Image Processing,2019,28(7):3312-3327.
[30]SUN D,YANG X,LIU M Y,et al.Pwc-net:Cnns for optical flow using pyramid,warping,and cost volume[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:8934-8943.
[31]NAH S,BAIK S,HONG S,et al.Ntire 2019 challenge on video deblurring and super-resolution:Dataset and study[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:1996-2005.
[32]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.

Related Articles 15

[1]	SUN Jie-qi, LI Ya-feng, ZHANG Wen-bo, LIU Peng-hui. Dual-field Feature Fusion Deep Convolutional Neural Network Based on Discrete Wavelet Transformation [J]. Computer Science, 2022, 49(6A): 434-440.
[2]	HU Fu-yuan, WAN Xin-jun, SHEN Ming-fei, XU Jiang-lang, YAO Rui, TAO Zhong-ben. Survey Progress on Image Instance Segmentation Methods of Deep Convolutional Neural Network [J]. Computer Science, 2022, 49(5): 10-24.
[3]	LENG Jia-xu, WANG Jia, MO Meng-jing-cheng, CHEN Tai-yue, GAO Xin-bo. Survey on Video Super-resolution Based on Deep Learning [J]. Computer Science, 2022, 49(2): 123-133.
[4]	YU Han-qing, YANG Zhen, YIN Zhi-jian. Tiny YOLOv3 Target Detection Algorithm Based on Region Activation Strategy [J]. Computer Science, 2021, 48(6A): 118-121.
[5]	SHI Xian-rang, SONG Ting-lun, TANG De-zhi, DAI Zhen-yong. Novel Deep Learning Algorithm for Monocular Vision:H_SFPN [J]. Computer Science, 2021, 48(4): 130-137.
[6]	XU Yu-jun, LI Chen. Light-weight Object Detection Network Optimized Based on YOLO Family [J]. Computer Science, 2021, 48(11A): 265-269.
[7]	XU Hua-jie, YANG Yang, LI Gui-lan. Material Recognition Method Based on Attention Mechanism and Deep Convolutional Neural Network [J]. Computer Science, 2021, 48(10): 220-225.
[8]	CAI Yu-han,XIONG Shu-hua,SUN Wei-heng,Karn PRADEEP,HE Xiao-hai. Video Compression Algorithm Combining Frame Rate Up-conversion with HEVC Standard Based on Motion Vector Refinement [J]. Computer Science, 2020, 47(2): 76-82.
[9]	QU Jia, SHI Zeng-lin, YE Yang-dong. Unbalanced Crowd Density Estimation Based on Convolutional Features [J]. Computer Science, 2018, 45(8): 236-241.
[10]	LEI Qian, HAO Cun-ming,ZHANG Wei-ping. Vehicle Recognition Based on Super-resolution and Deep Neural Networks [J]. Computer Science, 2018, 45(6A): 230-233.
[11]	CHEN Cheng, CHANG Kan, MO Cai-wang, LI Tian-yi and QIN Tuan-fa. Single Video Super-resolution Algorithm Based on Non-local Means and Total Variation Minimization [J]. Computer Science, 2018, 45(3): 263-267.
[12]	PAN Xiang and WANG Heng. Similar Character Recognition of License Plates Based on Deep Learning [J]. Computer Science, 2017, 44(Z6): 229-231.
[13]	WANG Wei, LIN Xi-jie and LI Xiao-qin. Improved Information Hiding Algorithm Based on Motion Estimation of H.264 [J]. Computer Science, 2015, 42(6): 151-157.
[14]	CHANG Jia-yi,QIN Rui,LI Qing,CHEN Xi and XU Jian-jun. Research of Obstacle Detection Based on Aerial Panorama Image [J]. Computer Science, 2014, 41(Z11): 78-82.
[15]	. Improved Method of Global Optical Flow Estimation [J]. Computer Science, 2012, 39(Z6): 510-512.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Video Super-resolution Method Based on Deep Learning Feature Warping

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 15

Metrics

Comments

Recommended 0