Computer Science

Computer Science ›› 2024, Vol. 51 ›› Issue (7): 197-205.doi: 10.11896/jsjkx.230400102

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Study on Algorithm of Depth Image Super-resolution Guided by High-frequency Information ofColor Images

LI Jiaying1, LIANG Yudong1,2, LI Shaoji1, ZHANG Kunpeng1, ZHANG Chao1,2   

  1. 1 School of Computer and Information Technology,Shanxi University,Taiyuan 030006,China
    2 Key Laboratory of Ministry of Education for Computation Intelligence and Chinese Information Processing,Shanxi University,Taiyuan 030006,China
  • Received:2023-04-16 Revised:2023-09-21 Online:2024-07-15 Published:2024-07-10
  • About author:LI Jiaying,born in 1998,master.Her main research interests include compu-ter vision and image processing.
    LIANG Yudong,born in 1988,Ph.D,associate professor,is a member of CCF(No.85977M).His main research interests include computer vision,image processing,and deep learning-based applications.
  • Supported by:
    National Natural Science Foundation of China(61802237,62272284),Fundamental Research Program of Shanxi Province(202203021221002,202203021211291),Natural Science Foundation of Shanxi Province,China(201901D211176,202103021223464),Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi(2019L0066),Science and Technology Major Project of Shanxi Province,China(202101020101019),Key R & D Program of Shanxi Province(202102070301019) and Special Fund for Science and Technology Innovation Teams of Shanxi(202204051001015).

PDF (PC)

478

Abstract: Depth image information is an important part of 3D scene information.However,due to the limitations of acquisition equipment and the diversity of imaging environments,the depth images acquired by depth sensors often have low resolution and less high-frequency information,which limits their further applications in various computer vision tasks.Depth image super-resolution attempts to improve the resolution of depth images and is a practical and valuable task.The RGB image in the same scene has high resolution and rich texture information,and some depth image super-resolution algorithms achieve significant improvement in algorithm performance by introducing RGB images from the same scene to provide guidance information.However,due to the structural inconsistency between RGB images and depth maps,how to utilize RGB information fully and effectively is still extremely challenging.To this end,this paper proposes a depth image super-resolution guided by high-frequency information of co-lor images.Specifically,a high-frequency feature extraction module is designed to adaptively learn high-frequency information of color images to guide the reconstruction of depth map edges.In addition,a feature self-attention module is designed to capture the global dependencies between features,extract deeper features to help recover details in the depth image.After cross-modal fusion,the depth image features and color image-guided features are reconstructed,and the proposed multi-scale feature fusion module is used to fuse the spatial structure information between different scale features to obtain reconstruction information including multi-level receptive fields.Finally,through the depth reconstruction module,the corresponding high-resolution depth map is recovered.Comprehensive qualitative and quantitative experimental results on public datasets have demonstrated that the proposed method outperforms comparative methods,which verifies its effectiveness.

Key words: Depth image super-resolution reconstruction, Deep learning, Cross-modal fusion, High-frequency information, Self-attention mechanism

CLC Number: 

  • TP391
[1]RICHARDT C,STOLL C,DODGSON N A,et al.Coherent spatiotemporal filtering,upsampling and rendering of RGBZ videos[C]//Computer Graphics Forum.Oxford,UK:Blackwell Publishing Ltd,2012:247-256.
[2]HE K,SUN J,TANG X.Guided image filtering[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2012,35(6):1397-1409.
[3]KOPF J,COHEN M F,LISCHINSKI D,et al.Joint bilateral upsampling[J].ACM Transactions on Graphics(ToG),2007,26(3):96-1-95-5.
[4]YANG Q,YANG R,DAVIS J,et al.Spatial-depth super resolution for range images[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2007:1-8.
[5]PARK J,KIM H,TAI Y W,et al.High quality depth map upsampling for 3D-TOF cameras[C]//2011 International Confe-rence on Computer Vision.IEEE,2011:1623-1630.
[6]FERSTL D,REINBACHER C,RANFTL R,et al.Image guideddepth upsampling using anisotropic total generalized variation[C]//Proceedings of the IEEE International Conference on Computer Vision.2013:993-1000.
[7]YE X,SUN B,WANG Z,et al.PMBANet:Progressive multi-branch aggregation network for scene depth super-resolution[J].IEEE Transactions on Image Processing,2020,29:7427-7442.
[8]JIANG Z,YUE H,LAI Y K,et al.Deep edge map guided depthsuper resolution[J].Signal Processing:Image Communication,2021,90:116040.
[9]GUO C,LI C,GUO J,et al.Hierarchical features driven residual learning for depth map super-resolution[J].IEEE Transactions on Image Processing,2018,28(5):2545-2557.
[10]YANG F,YANG H,FU J,et al.Learning texture transformer network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:5791-5800.
[11]CHEN Y,FAN H,XU B,et al.Drop an octave:Reducing spatial redundancy in convolutional neural networks with octave convolution[C]//Proceedings of the IEEE/CVF International Confe-rence on Computer Vision.2019:3435-3444.
[12]XIE J,FERIS R S,SUN M T.Edge-guided single depth image super resolution[J].IEEE Transactions on Image Processing,2015,25(1):428-438.
[13]FERSTL D,RUTHER M,BISCHOF H.Variational depth super resolution using example-based edge representations[C]//Proceedings of the IEEE International Conference on Computer Vision.2015:513-521.
[14]DONG C,LOY C C,HE K,et al.Learning a deep convolutional network for image super-resolution[C]//Computer Vision－ECCV 2014:13th European Conference,Zurich,Switzerland,September 6-12,2014,Proceedings,Part IV 13.Springer International Publishing,2014:184-199.
[15]DONG C,LOY C C,TANG X.Accelerating the super-resolution convolutional neural network[C]//Computer Vision－ECCV 2016:14th European Conference,Amsterdam,The Netherlands,October 11-14,2016,Proceedings,Part II 14.Springer International Publishing,2016:391-407.
[16]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[17]KIM J,LEE J K,LEE K M.Accurate image super-resolutionusing very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1646-1654.
[18]KIM J,LEE J K,LEE K M.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:1637-1645.
[19]RIEGLER G,RüTHER M,BISCHOF H.Atgv-net:Accuratedepth super-resolution[C]//Computer Vision－ECCV 2016:14th European Conference,Amsterdam,The Netherlands,October 11-14,2016,Proceedings,Part III 14.Springer International Publishing,2016:268-284.
[20]SONG X,DAI Y,QIN X.Deeply supervised depth map super-resolution as novel view synthesis[J].IEEE Transactions on Circuits and Systems for Video Technology,2018,29(8):2323-2336.
[21]SONG X,DAI Y,ZHOU D,et al.Channel attention based iterative residual learning for depth map super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:5631-5640.
[22]DIEBEL J,THRUN S.An application of markov random fields to range sensing[C]//Proceedings of the 18th International Conference on Neural Information Processing Systems.2005:291-298.
[23]GU S,ZUO W,GUO S,et al.Learning dynamic guidance fordepth image enhancement[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3769-3778.
[24]LI Y,HUANG J B,AHUJA N,et al.Deep joint image filtering[C]//Computer Vision－ECCV 2016:14th European Confe-rence,Amsterdam,The Netherlands,October 11-14,2016,Proceedings,Part IV 14.Springer International Publishing,2016:154-169.
[25]LI Y,HUANG J B,AHUJA N,et al.Joint image filtering with deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(8):1909-1923.
[26]SU H,JAMPANI V,SUN D,et al.Pixel-adaptive convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:11166-11175.
[27]HUI T W,LOY C C,TANG X.Depth map super-resolution by deep multi-scale guidance[C]//Computer Vision－ECCV 2016:14th European Conference,Amsterdam,The Netherlands,October 11-14,2016,Proceedings,Part III 14.Springer International Publishing,2016:353-369.
[28]LUTIO R,D'ARONCO S,WEGNER J D,et al.Guided super-resolution as pixel-to-pixel transformation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019:8829-8837.
[29]KIM B,PONCE J,HAM B.Deformable kernel networks forjoint image filtering[J].International Journal of Computer Vision,2021,129(2):579-600.
[30]HE L,ZHU H,LI F,et al.Towards fast and accurate real-world depth super-resolution:Benchmark dataset and baseline[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:9229-9238.
[31]SUN B,YE X,LI B,et al.Learning scene structure guidance via cross-task knowledge transfer for single depth super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:7792-7801.
[32]DENG X,DRAGOTTI P L.Deep convolutional neural network for multi-modal image restoration and fusion[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,43(10):3333-3348.
[33]ZHAO Z,ZHANG J,XU S,et al.Discrete cosine transform network for guided depth map super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:5697-5707.
[34]MALLICK A,ENGELHARDT A,BRAUN R,et al.Local Attention Guided Joint Depth Upsampling[C]//Vision,Modeling,and Visualization.The Eurographics Association,2022:135-1439.
[35]DONG J,PAN J,REN J S,et al.Learning spatially variant linearrepresentation models for joint filtering[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,44(11):8355-8370.
[36]ZHONG Z,LIU X,JIANG J,et al.Deep attentional guidedimage filtering[J].arXiv:2112.06401,2023.
[37]YANG Y,CAO Q,ZHANG J,et al.CODON:on orchestrating cross-domain attentions for depth super-resolution[J].International Journal of Computer Vision,2022,130(2):267-284.
[38]ZHOU C,ZHOU Q W,CHEN H M,et al.Recurrent Scale-by-scale Feature Fusion Network for RGBD Salient Object Detection[J].Journal of Chinese Computer Systems,2023,44(10):2276-2283.
[39]MARCHAND E,UCHIYAMA H,SPINDLER F.Pose estimation for augmented reality:a hands-on survey[J].IEEE Transactions on Visualization and Computer Graphics,2015,22(12):2633-2651.
[40]LU S,REN X,LIU F.Depth enhancement via low-rank matrix completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:3390-3397.
[41]HIRSCHMULLER H,SCHARSTEIN D.Evaluation of costfunctions for stereo matching[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2007:1-8.
[42]SCHARSTEIN D,PAL C.Learning conditional random fieldsfor stereo[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2007:1-8.
[1] YANG Heng, LIU Qinrang, FAN Wang, PEI Xue, WEI Shuai, WANG Xuan. Study on Deep Learning Automatic Scheduling Optimization Based on Feature Importance [J]. Computer Science, 2024, 51(7): 22-28.
[2] SHI Dianxi, GAO Yunqi, SONG Linna, LIU Zhe, ZHOU Chenlei, CHEN Ying. Deep-Init:Non Joint Initialization Method for Visual Inertial Odometry Based on Deep Learning [J]. Computer Science, 2024, 51(7): 327-336.
[3] FAN Yi, HU Tao, YI Peng. Host Anomaly Detection Framework Based on Multifaceted Information Fusion of SemanticFeatures for System Calls [J]. Computer Science, 2024, 51(7): 380-388.
[4] GAN Run, WEI Xianglin, WANG Chao, WANG Bin, WANG Min, FAN Jianhua. Backdoor Attack Method in Autoencoder End-to-End Communication System [J]. Computer Science, 2024, 51(7): 413-421.
[5] WANG Yingjie, ZHANG Chengye, BAI Fengbo, WANG Zumin. Named Entity Recognition Approach of Judicial Documents Based on Transformer [J]. Computer Science, 2024, 51(6A): 230500164-9.
[6] LIANG Fang, XU Xuyao, ZHAO Kailong, ZHAO Xuanfeng, ZHANG Guijun. Remote Template Detection Algorithm and Its Application in Protein Structure Prediction [J]. Computer Science, 2024, 51(6A): 230600225-7.
[7] PENG Bo, LI Yaodong, GONG Xianfu, LI Hao. Method for Entity Relation Extraction Based on Heterogeneous Graph Neural Networks and TextSemantic Enhancement [J]. Computer Science, 2024, 51(6A): 230700071-5.
[8] ZHANG Tianchi, LIU Yuxuan. Research Progress of Underwater Image Processing Based on Deep Learning [J]. Computer Science, 2024, 51(6A): 230400107-12.
[9] WANG Guogang, DONG Zhihao. Lightweight Image Semantic Segmentation Based on Attention Mechanism and Densely AdjacentPrediction [J]. Computer Science, 2024, 51(6A): 230300204-8.
[10] WANG Li, CHEN Gang, XIA Mingshan, HU Hao. DUWe:Dynamic Unknown Word Embedding Approach for Web Anomaly Detection [J]. Computer Science, 2024, 51(6A): 230300191-5.
[11] ZHANG Le, YU Ying, GE Hao. Mural Inpainting Based on Fast Fourier Convolution and Feature Pruning Coordinate Attention [J]. Computer Science, 2024, 51(6A): 230400083-9.
[12] WU Yibo, HAO Yingguang, WANG Hongyu. Rice Defect Segmentation Based on Dual-stream Convolutional Neural Networks [J]. Computer Science, 2024, 51(6A): 230600107-8.
[13] HOU Linhao, LIU Fan. Remote Sensing Image Fusion Combining Multi-scale Convolution Blocks and Dense Convolution Blocks [J]. Computer Science, 2024, 51(6A): 230400110-6.
[14] HUANG Yuanhang, BIAN Shan, WANG Chuntao. Gaussian Enhancement Module for Reinforcing High-frequency Details in Camera ModelIdentification [J]. Computer Science, 2024, 51(6A): 230700125-5.
[15] SUN Yang, DING Jianwei, ZHANG Qi, WEI Huiwen, TIAN Bowen. Study on Super-resolution Image Reconstruction Using Residual Feature Aggregation NetworkBased on Attention Mechanism [J]. Computer Science, 2024, 51(6A): 230600039-6.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.