计算机科学

计算机科学 ›› 2026, Vol. 53 ›› Issue (9): 228-239.doi: 10.11896/jsjkx.250700149

• 计算机图形学&多媒体 • 上一篇    下一篇

基于对偶特征补偿和光照增强的红外与可见光图像融合方法

李笑宇, 郝应光, 王洪玉   

  1. 大连理工大学信息与通信工程学院 辽宁 大连116024
  • 收稿日期:2025-07-22 修回日期:2025-11-14 出版日期:2026-09-15 发布日期:2026-09-10
  • 通讯作者: 郝应光(yghao@dlut.edu.cn)
  • 作者简介:(lxydut0813@163.com)
  • 基金资助:
    大连市科技创新基金(2022JJ11CG002)

Infrared and Visible Image Fusion Method Based on Dual Feature Compensation and IlluminationEnhancement

LI Xiaoyu, HAO Yingguang, WANG Hongyu   

  1. School of Information and Communication Engineering,Dalian University of Technology,Dalian,Liaoning 116024,China
  • Received:2025-07-22 Revised:2025-11-14 Published:2026-09-15 Online:2026-09-10
  • About author:LI Xiaoyu,born in 2000,postgraduate.Her main research interests include deep learning and image fusion.
    HAO Yingguang,born in 1968,associate professor.His main research interests include modeling complex time-varying systems and image processing algorithm.
  • Supported by:
    Dalian Science and Technology Innovation Fund(2022JJ11CG002).

PDF (PC)

312

摘要： 红外与可见光图像融合的目的是综合源图像中的信息,生成纹理细节丰富且满足高级视觉任务需求的融合图像。模态差异以及环境的影响,有可能导致融合图像纹理细节及对比度损失,进而影响下游图像处理任务。为此,提出了一个集成语义分割任务的基于对偶特征补偿和光照增强的红外与可见光图像自适应融合网络。首先,在特征提取部分,设计了基于状态空间模型(Mamba)的对偶特征补偿模块,促进了不同模态之间的信息交互。其次,利用循环神经网络(RNN),根据不同方向特征的贡献动态生成红外与可见光的融合权重,从而实现红外与可见光图像的自适应融合。此外,在网络前端设计了基于光照理论的图像增强模块对可见光图像进行增强,为后续融合过程提供细节更丰富的输入特征。训练阶段,使用融合和分割交替训练的方式,实现两任务的协同优化。在 MSRS,TNO 和 RoadScene这3个数据集上对所提模型的融合性能进行了评估,并在 MSRS 数据集上测试了其语义分割效果。实验结果表明,与当前主流的融合方法相比,该模型在主观效果和客观评价指标上都取得了较好的结果。

关键词: 对偶特征补偿, 自适应融合, 红外与可见光图像融合, 语义分割, 光照理论

Abstract: Infrared and visible image fusion aims to integrate complementary information from both modalities to produce a fused image with rich texture details suitable for high-level vision tasks.However,modality discrepancies and environmental factors often lead to texture loss and reduced contrast,hindering downstream performance.To address this issue,this paper proposes an adaptive fusion network that jointly performs image fusion and semantic segmentation,incorporating dual feature compensation and illumination enhancement.Specifically,a dual feature compensation module based on the state space model(Mamba) is designed in the feature extraction stage to enhance cross-modal information interaction.Then,a recurrent neural network(RNN) is used to dynamically generate fusion weights for infrared and visible features according to their directional contributions,enabling adaptive fusion.Additionally,a front-end illumination enhancement module improves visible image quality to enrich feature representation.During training,an alternating training strategy is adopted for the fusion and segmentation tasks to achieve joint optimization. Fusion performance is evaluated on the MSRS,TNO,and RoadScene datasets,while segmentation performance is assessed on MSRS.Results demonstrate that the proposed model outperforms mainstream infrared and visible image fusion methods in both subjective quality and objective metrics.

Key words: Dual feature compensation, Adaptive fusion, Infrared and visible image fusion, Semantic segmentation, Illumination theory

中图分类号: 

  • TP391
[1] ZHANG H,XU H,TIAN X,et al.Image fusion meets deeplearning:A survey and perspective[J].Information Fusion,2021,76:323-336.
[2] YI X,MA Y,LI Y,et al.Artificial intelligence facilitates information fusion for perception in complex environments[J].The Innovation,2025,6(4):10814-10817
[3] JAIN D K,ZHAO X,GONZÁLEZ-ALMAGRO G,et al.Multimodal pedestrian detection using metaheuristics with deep convolutional neural network in crowded scenes[J].Information Fusion,2023,95:401-414.
[4] ZHANG P,WANG D,LU H,et al.Learning adaptive attribute-driven representation for real-time RGB-T tracking[J].International Journal of Computer Vision,2021,129:2714-2729.
[5] HA Q,WATANABE K,KARASAWA T,et al.MFNet:To-wards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C] //2017 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).IEEE,2017:5108-5115.
[6] PARAMANANDHAM N,RAJENDIRAN K.Infrared and visible image fusion using discrete cosine transform and swarm intelligence for surveillance applications[J].Infrared Physics & Technology,2018,88:13-22.
[7] LIU J,WU G,LIU Z,et al.Infrared and Visible Image Fusion:From Data Compatibility to Task Adaption[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2025,47(4):2349-2369.
[8] CHEN J,LI X,LUO L,et al.Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J].Information Sciences,2020,508:64-78.
[9] LI H,WU X J,KITTLER J.MDLatLRR:A novel decomposition method for infrared and visible image fusion[J].IEEE Transactions on Image Processing,2020,29:4733-4746.
[10] MA J,ZHOU Z,WANG B,et al.Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J].Infrared Physics & Technology,2017,82:8-17.
[11] FU Z,WANG X,XU J,et al.Infrared and visible images fusion based on RPCA and NSCT[J].Infrared Physics & Technology,2016,77:114-123.
[12] LIU Y,CHEN X,CHENG J,et al.A medical image fusionmethod based on convolutional neural networks[C] //2017 20th International Conference on Information Fusion(Fusion).IEEE,2017:1-7.
[13] ZHANG Y,LIU Y,SUN P,et al.IFCNN:A general image fusion framework based on convolutional neural network[J].Information Fusion,2020,54:99-118.
[14] ZHAO Z,BAI H,ZHANG J,et al.Cddfuse:Correlation-drivendual-branch feature decomposition for multi-modality image fusion[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:5906-5916.
[15] LIU Y,TIAN Y,ZHAO Y,et al.Vmamba:Visual state space model[J].Advances in Neural Information Processing Systems,2024,37:103031-103063.
[16] LI H,WU X J.DenseFuse:A fusion approach to infrared and visible images[J].IEEE Transactions on Image Processing,2018,28(5):2614-2623.
[17] LI H,WU X J,KITTLER J.RFN-Nest:An end-to-end residual fusion network for infrared and visible images[J].Information Fusion,2021,73:72-86.
[18] TANG L,YUAN J,ZHANG H,et al.PIAFusion:A progressive infrared and visible image fusion network based on illumination aware[J].Information Fusion,2022,83:79-92.
[19] YANG Y.Multimodal medical image fusion through a newDWT based technique[C] //2010 4th International Conference on Bioinformatics and Biomedical Engineering.IEEE,2010:1-4.
[20] MA J,ZHOU Z,WANG B,et al.Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J].Infrared Physics & Technology,2017,82:8-17.
[21] CVEJIC N,BULL D,CANAGARAJAH N.Region-based multimodal image fusion using ICA bases[J].IEEE Sensors Journal,2007,7(5):743-751.
[22] FU Z,WANG X,XU J,et al.Infrared and visible images fusion based on RPCA and NSCT[J].Infrared Physics & Technology,2016,77:114-123.
[23] MOU J,GAO W,SONG Z.Image fusion based on non-negative matrix factorization and infrared feature extraction[C] //2013 6th International Congress on Image and Signal Processing(CISP).IEEE,2013,2:1046-1050.
[24] LI H,XU T,WU X J,et al.Lrrnet:A novel representation learning guided fusion network for infrared and visible images[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2023,45(9):11040-11052.
[25] LONG Y,JIA H,ZHONG Y,et al.RXDNFuse:A aggregatedresidual dense network for infrared and visible image fusion[J].Information Fusion,2021,69:128-141.
[26] MA J,YU W,LIANG P,et al.FusionGAN:A generative adversarial network for infrared and visible image fusion[J].Information Fusion,2019,48:11-26.
[27] MA J,ZHANG H,SHAO Z,et al.GANMcC:A generative adversarial network with multiclassification constraints for infrared and visible image fusion[J].IEEE Transactions on Instrumentation and Measurement,2020,70:1-14.
[28] MA J,TANG L,FAN F,et al.SwinFusion:Cross-domain long-range learning for general image fusion via swin transformer[J].IEEE/CAA Journal of Automatica Sinica,2022,9(7):1200-1217.
[29] TANG W,HE F,LIU Y,et al.DATFuse:Infrared and visible image fusion via dual attention transformer[J].IEEE Transactions on Circuits and Systems for Video Technology,2023,33(7):3159-3172.
[30] TANG L,YUAN J,MA J.Image fusion in the loop of high-level vision tasks:A semantic-aware real-time infrared and visible image fusion network[J].Information Fusion,2022,82:28-42.
[31] LIU J,FAN X,HUANG Z,et al.Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:5802-5811.
[32] LIU J,LIU Z,WU G,et al.Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation[C] //Proceedings of the IEEE/CVF International Conference on Computer Vision.2023:8115-8124.
[33] ZHANG H,ZUO X,JIANG J,et al.Mrfs:Mutually reinforcing image fusion and segmentation[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2024:26974-26983.
[34] GU A,JOHNSON I,GOEL K,et al.Combining recurrent,con-volutional,and continuous-time models with linear state space layers[J].Advances in Neural Information Processing Systems,2021,34:572-585.
[35] BERMAN M,TRIKI A R,BLASCHKO M B.The lovász-softmax loss:A tractable surrogate for the optimization of the intersection-over-union measure in neural networks[C] //Procee-dings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:4413-4421.
[36] TOET A,HOGERVORST M A.Progress in color night vision[J].Optical Engineering,2012,51(1):010901.
[37] XU H,MA J,LE Z,et al.Fusiondn:A unified densely connected network for image fusion[C] //Proceedings of the AAAI Conference on Artificial Intelligence.2020:12484-12491.
[38] XU H,MA J,JIANG J,et al.U2Fusion:A unified unsupervised image fusion network[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,44(1):502-518.
[39] ZHANG H,MA J.SDNet:A versatile squeeze-and-decomposi-tion network for real-time image fusion[J].International Journal of Computer Vision,2021,129(10):2761-2785.
[40] TANG L,DENG Y,MA Y,et al.SuperFusion:A versatileimage registration and fusion network with semantic awareness[J].IEEE/CAA Journal of Automatica Sinica,2022,9(12):2121-2137.
[41] ROBERTS J W,VAN AARDT J A,AHMED F B.Assessment of image fusion procedures using entropy,image quality,and multispectral classification[J].Journal of Applied Remote Sen-sing,2008,2(1):023522.
[42] QU G,ZHANG D,YAN P.Information measure for perform-ance of image fusion[J].Electronics Letters,2002,38(7):313-315.
[43] CUI G,FENG H,XU Z,et al.Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J].Optics Communications,2015,341:199-209.
[44] HAN Y,CAI Y,CAO Y,et al.A new image fusion performance metric based on visual information fidelity[J].Information Fusion,2013,14(2):127-135.
[45] WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[46] XYDEAS C S,PETROVIC V.Objective image fusion perfor-mance measure[J].Electronics Letters,2000,36(4):308-309.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市两江新区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发