Computer Science

Computer Science ›› 2025, Vol. 52 ›› Issue (6): 228-238.doi: 10.11896/jsjkx.241200092

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Research on Depth Image Super-resolution Algorithm for High and Low Frequency Feature Modulation Fusion Guided by Color Images

XU Hanzhi1, LI Jiaying2, LIANG Yudong2,3, WEI Wei2,3   

  1. 1 School of Mathematical Science,Shanxi University,Taiyuan 030006,China
    2 School of Computer and Information Technology,Shanxi University,Taiyuan 030006,China
    3 Key Laboratory of Ministry of Education for Computation Intelligence and Chinese Information Processing,Shanxi University,Taiyuan 030006,China
  • Received:2024-12-11 Revised:2025-03-25 Online:2025-06-15 Published:2025-06-11
  • About author:XU Hanzhi,born in 2004,undergra-duate.His main research interests include computer vision and image processing.
    LIANG Yudong,born in 1988,Ph.D,as-sociate professor,is a member of CCF(No.85977M).His main research interests include computer vision,image processing and deep learning-based applications.
  • Supported by:
    National Natural Science Foundation of China(61802237,62272284),Fundamental Research Program of Shanxi Province(202203021221002,202203021211291),Natural Science Foundation of Shanxi Province(201901D211176,202103021223464),Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi(2019L0066),Science and Technology Major Project of Shanxi Province(202101020101019),Key R&D Program of Shanxi Provine(202102070301019) and Special Fund for Science and Technology Innovation Teams of Shanxi(202204051001015).

PDF (PC)

61

Abstract: Depth images effectively describe the information of a 3D scene.However,the acquisition equipment and imaging environment limit the resolution and high-frequency information of the depth images acquired by depth sensors.It is imperative to improve the resolution of depth images.Some depth map super-resolution algorithms have significantly improved their performance by introducing RGB images from the same scene to provide guidance information for the depth map super-resolution process.The key challenge lies in effectively leveraging the RGB information to guide the depth map super-resolution reconstruction process,addressing the modal inconsistency between the depth map and RGB images.Existing methods primarily focus on high-frequency information,overlooking the low-frequency global information crucial for algorithm performance.To address these limitations,this paper proposes a novel color image-guided,high and low-frequency feature modulation fusion super-resolution reconstruction algorithm for depth maps.A two-branch feature extraction module extracts high and low frequency features from color and depth images,respectively.CNN and Transformer are used in each branch to extract local high frequency and global low frequency information.A two-way transformation and fusion between high frequency information and low frequency information of color and depth images is achieved by constructing a two-way modulation module.The model fully exploits the complementary information between the depth image and the color image.It uses a bidirectional modulation within different modes and different frequencies and the subsequent fusion of high and low-frequency information.The depth super-resolution algorithm based on the guidance of the color image can achieve better reconstruction results.The lossless information compression using reversible neural network INN extracts high-frequency detail information more effectively,and the quadtree attention mechanism reduces the computational complexity of the Transformer in extracting global information,improving the efficiency of the algorithm.The experimental results on the public datasets show that the proposed method outperforms the comparison methods in both quantitative and qualitative aspects,achieving better subjective visualization results.

Key words: Depth image super-resolution reconstruction, Hybrid features, Bidirectional modulation, Quadtree attention mechanics

CLC Number: 

  • TP391
[1]HAN X F,LAGA H,BENNAMOUN M.Image-Based 3D Object Reconstruction:State-of-the-Art and Trends in the Deep Learning Era[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,43(5):1578-1604.
[2]MEULEMAN A,BAEK S H,HEIDE F,et al.Single-Shot Monocular RGB-D Imaging Using Uneven Double Refraction[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2020:2462-2471.
[3]WANG F,PAN J,XU S,et al.Learning Discriminative Cross-Modality Features for RGB-D Saliency Detection[J].IEEE Transactions on Image Processing,2022,31:1285-1297.
[4]HE K,SUN J,TANG X.Guided Image Filtering[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2013,35(6):1397-1409.
[5]RICHARDT C,STOLL C,DODGSON N A,et al.CoherentSpatiotemporal Filtering,Upsampling and Rendering of RGBZVideos[J].Computer Graphics Forum,2012,31(2),247-256.
[6]HE L,ZHU H,LI F,et al.Towards Fast and Accurate Real-World Depth Super-Resolution:Benchmark Dataset and Baseline[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2021:9225-9234.
[7]DONG X,YOKOYA N,WANG L,et al.Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution[C]//European Conference on Computer Vision.2022:1-18.
[8]TANG S,ZHANG J,ZHU S,et al.Quadtree Attention for Vision Transformers[J].arXiv:2201.02767,2022.
[9]HAN Y B,SHU F,SUN J T,et al.Image Super-Resolution Reconstruction Based on Mg-Gmres Algorithm[J].Chinese Journal of Computers,2007,30(6):1028-1034.
[10]ŠROUBEK F,FLUSSER J,CRISTÓBAL G.Super-Resolutionand Blind Deconvolution for Rational Factors With an Application to Color Images[J].The Computer Journal,2009,52(1):142-152.
[11]SCHWARZ S,SJÖSTRÖM M,OLSSON R.A Weighted Optimization Approach to Time-of-Flight Sensor Fusion[J].IEEE Transactions on Image Processing,2014,23(1):214-225.
[12]CHEN X X,QI C.Single-Image Super-Resolution Via Low-Rank Matrix Recovery and Joint Learning[J].Chinese Journal of Computers,2014,37(6):1372-1379.
[13]KHODDAMI A A,MOALLEM P,KAZEMI M.Depth Map Super Resolution Using Structure-Preserving Guided Filtering[J].IEEE Sensors Journal,2022,22(13):13144-13152.
[14]YOON Y,JEON H G,YOO D,et al.Learning A Deep Convolutional Network for Light-Field Image Super-Resolution[C]//2015 IEEE International Conference on Computer Vision Workshop(ICCVW).2015:57-65.
[15]ZENG K,YU J,WANG R,et al.Coupled Deep Autoencoder for Single Image Super-Resolution[J].IEEE Transactions on Cybernetics,2017,47:27-37.
[16]WEI Y,YUAN Q,SHEN H,et al.Boosting the Accuracy ofMultispectral Image Pansharpening by Learning A Deep Residual Network[J].IEEE Geoscience and Remote Sensing Letters,2017,14(10):1795-1799.
[17]LU B,LING Q.Edge-Guided Depth Image Super-ResolutionBased on KSVD[J].IEEE Access,2020,8:41108-41115.
[18]JAMES D,SEBASTIAN T.An Application of Markov Random Fields to Range Sensing[C]//Proceedings of the 19th International Conference on Neural Information Processing Systems.MIT,2005:291-298.
[19]LIU X,ZHAI D,CHEN R,et al.Depth Super-Resolution ViaJoint Color-Guided Internal and External Regularizations[J].IEEE Transactions on Image Processing,2019,28(4):1636-1645.
[20]WANG H,YANG M,LAN X,et al.Depth Map Recovery Based on A Unified Depth Boundary Distortion Model[J].IEEE Transactions on Image Processing,2022,31:7020-7035.
[21]GUO C,LI C,GUO J,et al.Hierarchical Features Driven Resi-dual Learning for Depth Map Super-Resolution[J].IEEE Transactions on Image Processing,2019,28(5):2545-2557.
[22]LI Y,HUANG J B,AHUJA N,et al.Joint Image Filtering with Deep Convolutional Networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(8):1909-1923.
[23]LI Y,HUANG J B,AHUJA N,et al.Deep Joint Image Filtering[C]//European Conference on Computer Vision.2016:154-169.
[24]SU H,JAMPANI V,SUN D,et al.Pixel-Adaptive Convolutio-nal Neural Networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2019:11158-11167.
[25]HUI T W,LOY C C,TANG X.Depth Map Super-Resolution by Deep Multi-Scale Guidance[C]//European Conference on Computer Vision.2016:353-369.
[26]ZHAO Z,ZHANG J,XU S,et al.Discrete Cosine TransformNetwork for Guided Depth Map Super-Resolution[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2021:5687-5697.
[27]MIAO Y,ZHANG X,SUN Y,et al.Ca-DBMNet:a channel attention based dual branch multi-scale network for depth map super-resolution[J].Multimedia Tools and Applications,2025,84:1577-1596.
[28]LI J,LIANG Y,LI S,et al.Study on Algorithm of Depth Image Super-Resolution Guided by High-Frequency Information of Color Images[J].Computer Science,2024,51(7):197-205.
[29]WANG Z,YAN Z,YANG J.SGNet:structure guided networkvia gradient-frequency awareness for depth map super-resolution[C]//Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence.AAAI,2024:5823-5831.
[30]WANG Z,YAN Z,YANG M H,et al.Scene Prior Filtering for Depth Map Super-Resolution[J].arXiv:2402.13876,2024.
[31]YUAN J,JIANG H,LI X,et al.Recurrent structure attentionguidance for depth super-resolution[C]//Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence.AAAI,2023:3331-3339.
[32]ZHANG C,WANG L,CHENG S.HCGNet:A Hybrid Change Detection Network Based on CNN and GNN[J].IEEE Transactions on Geoscience and Remote Sensing,2024,62:1-12.
[33]ZHONG Z,LIU X,JIANG J,et al.Deep Attentional GuidedImage Filtering[J].IEEE Transactions on Neural Networks and Learning Systems,2024,35(9):12236-12250.
[34]ZAMIR S W,ARORA A,KHAN S H,et al.Restormer:Effi-cient Transformer for High-Resolution Image Restoration[J].arXiv:2111.09881,2021.
[35]LI K,YU R,WANG Z,et al.Locality Guidance for Improving Vision Transformers on Tiny Datasets[C]//European Confe-rence on Computer Vision.2022:110-127.
[36]ZHOU M,FU X,HUANG J,et al.Effective Pan-SharpeningWith Transformer and Invertible Neural Network[J].IEEE Transactions on Geoscience and Remote Sensing,2022,60:1-15.
[37]WU H,XIAO B,CODELLA N C F,et al.CvT:Introducing Convolutions to Vision Transformers[C]//2021 IEEE/CVF International Conference on Computer Vision(ICCV).IEEE,2021:22-31.
[38]SEIF G,ANDROUTSOS D.Edge-Based Loss Function for Single Image Super-Resolution[C]//2018 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).2018:1468-1472.
[39]MARCHAND E,UCHIYAMA H,SPINDLER F.Pose Estima-tion for Augmented Reality:A Hands-on Survey[J].IEEE Transactions on Visualization and Computer Graphics,2016,22(12):2633-2651.
[40]LU S,REN X,LIU F.Depth Enhancement Via Low-Rank Matrix Completion[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition.2014:3390-3397.
[41]HIRSCHMULLER H,SCHARSTEIN D.Evaluation of CostFunctions for Stereo Matching[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition.2007:1-8.
[42]SCHARSTEIN D,PAL C.Learning Conditional Random Fields for Stereo[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition.2007:1-8.
[43]KIM B,PONCE J,HAM B.Deformable Kernel Networks forJoint Image Filtering[J].International Journal of Computer Vision,2021,129(2):579-600.
[44]DENG X,DRAGOTTIP L.Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,43(10):3333-3348.
[45]DONG J,PAN J,REN J S,et al.Learning Spatially Variant Li-near Representation Models for Joint Filtering[J].IEEE Tran-sactions on Pattern Analysis and Machine Intelligence,2022,44(11):8355-8370.
[1] LI Jiaying, LIANG Yudong, LI Shaoji, ZHANG Kunpeng, ZHANG Chao. Study on Algorithm of Depth Image Super-resolution Guided by High-frequency Information ofColor Images [J]. Computer Science, 2024, 51(7): 197-205.
[2] GENG Huan, Tong CAI ,Qing Sheng （Department of Computer Science gc Technology, University of Science ＆ Technology of China, Hefei 230027）. [J]. Computer Science, 2006, 33(6): 52-54.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.