计算机科学 ›› 2023, Vol. 50 ›› Issue (2): 190-200.doi: 10.11896/jsjkx.220100074

• 计算机图形学&多媒体 • 上一篇    下一篇

红外和可见光图像融合算法的研究进展

魏琦1,2, 赵娟1   

  1. 1 中国科学院深圳先进技术研究院 广东 深圳 518055
    2 中国科学院大学 北京 100039
  • 收稿日期:2022-01-09 修回日期:2022-07-13 出版日期:2023-02-15 发布日期:2023-02-22
  • 通讯作者: 赵娟(juan.zhao@siat.ac.cn)
  • 作者简介:(qi.wei@sait.ac.cn)
  • 基金资助:
    国防科技创新特区项目(20-163-00-KX-001-002-02,20-163-00-KX-001-003-02)

Research Progress of Infrared and Visible Image Fusion Algorithms

WEI Qi1,2, ZHAO Juan1   

  1. 1 Shenzhen Institute of Advanced Technology,Chinese Academy of Sciences,Shenzhen,Guangdong 518055,China
    2 University of Chinese Academy of Sciences,Beijing 100039,China
  • Received:2022-01-09 Revised:2022-07-13 Online:2023-02-15 Published:2023-02-22
  • Supported by:
    National Defense Science and Technology Innovation Special Zone Project(20-163-00-KX-001-002-02,20-163-00-KX-001-003-02)

摘要: 红外图像便于识别热源目标,可见光图像包含丰富的纹理信息。红外和可见光的融合图像兼顾了两个波段传感器的优势,可以清楚地显示热源目标及其背景,在军事侦察、安防监控、遥感监测等领域有着广泛的应用,已成为图像融合领域的重点研究方向。近年来,国内外学者对红外和可见光图像融合算法开展了大量研究。文中首先对现有的图像融合算法进行了详细介绍,包括多尺度变换、稀疏表示的传统图像处理方法和基于CNN,GAN,AE这3种常见网络结构的深度学习图像融合算法。接着综述了融合图像的评价方法,对常见的多种客观评价指标进行了归类分析。然后开展对比实验,对各种方法进行了主观评价和定量分析,指出不同方法的优势和不足。最后,对红外和可见光图像融合技术的未来发展趋势进行展望。

关键词: 图像融合, 红外图像, 可见光图像, 神经网络, 评价准则

Abstract: Infrared images are easy to identify thermal targets,and visible images have rich texture information.The fusion of infrared and visible images takes the advantages of both optical bands which can clearly show the targets and background.It has been widely used in many fields such as military reconnaissance,security monitoring,remote sensing measurement,and becomes a key research direction in the field of image fusion.In recent years,infrared and visible image fusion algorithms have attracted the attention of researchers around the world and have been studied abundantly.In this paper,the image fusion algorithms are introduced firstly,including traditional image processing methods of multi-scale transformation,sparse representation,and deep lear-ning algorithms based on CNN,GAN,AE.Then,the evaluation methods of fusion images are summarized,and a variety of common objective evaluation indexes are classified.After that,comparative experiments are carried out to subjectively evaluate and quantitatively analyze the advantages and disadvantages of these algorithms.Finally,the development trend of infrared and visible image fusion methods is prospected.

Key words: Image fusion, Infrared image, Visible image, Neural network, Evaluation metric

中图分类号: 

  • TP391
[1]LI S,KANG X,FANG L,et al.Pixel-level image fusion:A survey of the state of the art[J].information Fusion,2017,33:100-112.
[2]MA J,MA Y,LI C.Infrared and visible image fusion methods and applications:A survey[J].Information Fusion,2019,45:153-178.
[3]HOGERVORST M,TOET A.Fast natural color mapping fornight-time imagery[J].Information Fusion,2010,11(2):69-77.
[4]TOET A.Natural colour mapping for multiband nightvisionimagery[J].Information Fusion,2003,4(3):155-166.
[5]HAN J,BHANU B.Fusion of color and infrared video for mo-ving human detection[J].Pattern Recognition,2007,40(6):1771-1784.
[6]SINGH R,VATSA M,NOORE A.Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition[J].Pattern Recognition,2008,41(3):880-893.
[7]BHATNAGAR G,LIU Z.A novel image fusion framework for night-vision navigation and surveillance[J].Signal,Image and Video Processing,2015,9(1):165-175.
[8]PARAMANANDHAM N,RAJENDIRAN K.Multi sensorimage fusion for surveillance applications using hybrid image fusion algorithm[J].Multimedia Tools and Applications,2018,77(10):12405-12436.
[9]BULANON D,BURKS T,ALCHANATIS V.Image fusion ofvisible and thermal images for fruit detection[J].Biosystems engineering,2009,103(1):12-22.
[10]SANCHEZ V,PRINCE G,CLARKSON JP,et al.Registration of thermal and visible light images of diseased plants using silhouette extraction in the wavelet domain[J].Pattern Recognition,2015,48(7):2119-2128.
[11]SIMONE G,FARINA A,MORABITO F C,et al.Image fusiontechniques for remote sensing applications[J].Information fusion,2002,3(1):3-15.
[12]LI H,DING W,CAO X,et al.Image registration and fusion of visible and infrared integrated camera for medium-altitude unmanned aerial vehicle remote sensing[J].Remote Sensing,2017,9(5):441.
[13]BURT P J,ADELSON E H.The Laplacian Pyramid as a Compact Image Code[J].IEEE Transactions on Communications,1983,31(4):532-540.
[14]TOET A,VAN RUYVEN L J,VALETON J M.Merging thermal and visual images by a contrast pyramid[J].Optical Engineering,1989,28(7):287789.
[15]ZHANG Z,BLUM R S.A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application[C]//Proceedings of the IEEE.1999:1315-1326.
[16]CHEN J,LI X,LUO L,et al.Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J].Information Sciences,2020,508:64-78.
[17]LI S,KANG X,HU J.Image fusion with guided filtering[J].IEEE Trans Image Process,2013,22(7):2864-2875.
[18]ZHANG Q,LIU Y,BLUM R S,et al.Sparse representationbased multi-sensor image fusion for multi-focus and multi-modality images:A review[J].Information Fusion,2018,40:57-75.
[19]WANG K,QI G,ZHU Z,et al.A novel geometric dictionary construction approach for sparse representation based image fusion[J].Entropy,2017,19(7):306.
[20]YIN H.Sparse representation with learned multiscale dictionary for image fusion[J].Neurocomputing,2015,148:600-610.
[21]SUN C,ZHANG C,XIONG N.Infrared and Visible Image Fusion Techniques Based on Deep Learning:A Review[J].Electronics,2020,9(12):2162.
[22]MA J,YU W,LIANG P,et al.FusionGAN:A generative adversarial network for infrared and visible image fusion[J].Information Fusion,2019,48:11-26.
[23]PRABHAKAR K R,SRIKAR V S,BABU R V.Deepfuse:Adeep unsupervised approach for exposure fusion with extreme exposure image pairs[C]//Proceedings of the IEEE Interna-tional Conference on Computer Vision.2017:4714-4722.
[24]LIU Y,CHEN X,CHENG J,et al.Infrared and visible image fusion with convolutional neural networks[J].International Journal of Wavelets,Multiresolution and Information Processing,2018,16(3):1850018.
[25]LI H,WU X J,KITTLER J.RFN-Nest:An end-to-end residual fusion network for infrared and visible images[J].Information Fusion,2021,73:72-86.
[26]HE D X,MENG Y,WANG C Y.Contrast pyramid based imagefusion scheme for infrared image and visible image[C]//2011 IEEE International Geoscience and Remote Sensing Sympo-sium.2011:597-600.
[27]TOET A.A morphological pyramidal image decomposition[J].Pattern Recognition Letters,1989,9(4):255-261.
[28]GROSSMANN A,MORLET J.Decomposition of Hardy functions into square integrable wavelets of constant shape[J].SIAM Journal on Mathematical Analysis,1984,15(4):723-736.
[29]ZHAN L,ZHUANG Y,HUANG L.Infrared and visible images fusion method based on discrete wavelet transform[J].Journal of Computers,2017,28(2):57-71.
[30]HAN X,LI Z L,YAO D L,et al.Fusion of infrared and visible images based on discrete wavelet transform[C]//The Photoelec-tronic Technology Committee Conferences.2015:387-392.
[31]SELESNICK I W,BARANIUK R G,KINGSBURY N C.The dual-tree complex wavelet transform[J].IEEE Signal Processing Magazine,2005,22(6):123-151.
[32]ZUO Y,LIU J,BAI G,et al.Airborne infrared and visible image fusion combined with region segmentation[J].Sensors,2017,17(5):1127.
[33]MINGHUI S,LU L,YUANXI P,et al.Infrared & visible images fusion based on redundant directional lifting-based wavelet and saliency detection[J].Infrared Physics & Technology,2019,101:45-55.
[34]ZOU Y,LIANG X,WANG T.Visible and infrared image fusion using the lifting wavelet[J].TELKOMNIKA Indonesian Journal of Electrical Engineering,2013,11(11):6290-6295.
[35]DO M N,VETTERLI M.Contourlets:a directional multiresolution image representation[C]//Proceedings of International Conference on Image Processing.2002.
[36]DA CUNHA A L,ZHOU J,DO M N.The nonsubsampled con-tourlet transform:theory,design,and applications[J].IEEE Transactions on Image Processing,2006,15(10):3089-3101.
[37]MENG F,SONG M,GUO B,et al.Image fusion based on object region detection and Non-Subsampled Contourlet Transform[J].Computers & Electrical Engineering,2017,62:375-383.
[38]KONG W,WANG B,LEI Y.Technique for infrared and visible image fusion based on non-subsampled shearlet transform and spiking cortical model[J].Infrared Physics & Technology,2015,71:87-98.
[39]CAO Y,LI X,CHEN X,et al.Research on Fusion Method of Infrared and Visible Image Based on NSST and Guided Filtering[J].Journal of Changchun University of Science and Technology(Natural Science Edition),2022,45(3):28-34.
[40]LIU Y,LIU S,WANG Z.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion,2015,24:147-164.
[41]LIU G T,ZHOU J H,LI T,et al.Infrared and visible image fusion through hybrid curvature filtering image decomposition[J].Infrared Physics & Technology,2022,120:103938.
[42]YANG B,LI S.Multifocus image fusion and restoration withsparse representation[J].IEEE Transactions on Instrumentation and Measurement,2009,59(4):884-892.
[43]LIU C H,QI Y,DING W R.Infrared and visible image fusion method based on saliency detection in sparse domain[J].Infrared Physics & Technology,2017,83:94-102.
[44]BIN Y,CHAO Y,GUO Y H.Efficient image fusion with approximate sparse representation[J].International Journal of Wavelets,Multiresolution and Information Processing,2016,14(4):1650024.
[45]AHARON M,ELAD M,BRUCKSTEIN A.K-SVD:An algo-rithm for designing overcomplete dictionaries for sparse representation[J].IEEE Transactions on Signal Processing,2006,54(11):4311-4322.
[46]PATI Y C,REZAIIFAR R,KRISHNAPRASAD P S.Orthogonal matching pursuit:Recursive function approximation with applications to wavelet decomposition[C]//Proceedings of 27th Asilomar Conference on Signals,Systems and Computers.1993:40-44.
[47]YANG B,LI S.Pixel-level image fusion with simultaneous orthogonal matching pursuit[J].Information Fusion,2012,13(1):10-19.
[48]LI H,LIU L,HUANG W,et al.An improved fusion algorithm for infrared and visible images based on multi-scale transform[J].Infrared Physics & Technology,2016,74:28-37.
[49]MITIANOUDIS N,ANTONOPOULOS S A,STATHAKI T.Region-based ICA image fusion using textural information[C]//2013 18th International Conference on Digital Signal Processing(DSP).2013:1-6.
[50]KONG W,LEI Y,ZHAO H.Adaptive fusion method of visible light and infrared images based on non-subsampled shearlet transform and fast non-negative matrix factorization[J].Infrared Physics & Technology,2014,67:161-172.
[51]CAI J J,CHENG Q M,PENG M J,et al.Fusion of infrared and visible images based on nonsubsampled contourlet transform and sparse K-SVD dictionary learning[J].Infrared Physics & Technology,2017,82:85-95.
[52]CHEN G,CHEN Y,LI J,et al.Infrared and visible image fusionbased on discrete nonseparable shearlet transform and convolutional sparse representation[J].Journal of Jilin University(Engineering and Technology Edition),2021,51(3):996-1010.
[53]LIU Y,LIU S P,WANG Z F.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion,2015,24:147-164.
[54]ZHANG X Y,MA Y,FAN F,et al.Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition[J].Journal of the Optical Society of America A-Optics Image Science and Vision,2017,34(8):1400-1410.
[55]HOU R C,ZHOU D M,NIE R C,et al.Infrared and VisibleImages Fusion Using Visual Saliency and Dual-PCNN[J].Com-puter Science,2018,45(6A):162-166.
[56]LIU Y,CHEN X,WARD R K,et al.Image fusion with convolutional sparse representation[J].IEEE Signal Processing Letters,2016,23(12):1882-1886.
[57]LIU Y,CHEN X,PENG H,et al.Multi-focus image fusion with a deep convolutional neural network[J].Information Fusion,2017,36:191-207.
[58]LI H,WU X J,KITTLER J.Infrared and visible image fusionusing a deep learning framework[C]//2018 24th International Conference on Pattern Recognition(ICPR).2018:2705-2710.
[59]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[60]LI H,WU X J,DURRANI T S.Infrared and visible image fusion with ResNet and zero-phase component analysis[J].Infrared Physics & Technology,2019,102:103039.
[61]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[62]KESSY A,LEWIN A,STRIMMER K.Optimal whitening anddecorrelation[J].The American Statistician,2018,72(4):309-314.
[63]ZHANG Y,LIU Y,SUN P,et al.IFCNN:A general image fusion framework based on convolutional neural network[J].Information Fusion,2020,54:99-118.
[64]XU H,MA J,LE Z,et al.Fusiondn:A unified densely connected network for image fusion[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:12484-12491.
[65]HUANG G,LIU Z,VAN DER MAATEN L,et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4700-4708.
[66]XU H,MA J,JIANG J,et al.U2Fusion:A Unified Unsuper-vised Image Fusion Network[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,44(1):502-518.
[67]LI C,HOU J,LI J,et al.Infrared and Visible Image Fusion Method Based on Attention and Residual Concatenation[J].Computer Engineering,2022,48(7):234-240.
[68]MA J,LIANG P,YU W,et al.Infrared and visible image fusion via detail preserving adversarial learning[J].Information Fusion,2020,54:85-98.
[69]YUAN C,SUN C,TANG X,et al.FLGC-Fusion GAN:An Enhanced Fusion GAN Model by Importing Fully Learnable Group Convolution[J].Mathematical Problems in Engineering,2020,2020:6384831.
[70]LI Q,LU L,LI Z,et al.Coupled GAN With Relativistic Discri-minators for Infrared and Visible Images Fusion[J].IEEE Sensors Journal,2021,21(6):7458-7467.
[71]TANG L L,LIU G,XIAO G,et al.p Infrared and visible image fusion based on guided hybrid model and generative adversarial network[J].Infrared Physics & Technology,2022,120:103914.
[72]LI H,WU X J.DenseFuse:A fusion approach to infrared andvisible images[J].IEEE Transactions on Image Processing,2018,28(5):2614-2623.
[73]LI H,WU X J,DURRANI T.NestFuse:An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models[J].IEEE Transactions on Instrumentation and Measurement,2020,69(12):9645-9656.
[74]ZHOU Z,SIDDIQUEE M M R,TAJBAKHSH N,et al.UNet++:A Nested U-Net Architecture for Medical Image Segmentation[C]//Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support.2018:3-11.
[75]FU Y,WU X J.A dual-branch network for infrared and visible image fusion[C]//2020 25th International Conference on Pattern Recognition(ICPR).2021:10675-10680.
[76]XU H,ZHANG H,MA J.Classification Saliency-Based Rulefor Visible and Infrared Image Fusion[J].IEEE Transactions on Computational Imaging,2021,7:824-836.
[77]ZHANG L,LI H,ZHU R,et al.An infrared and visible image fusion algorithm based on ResNet-152[J].Multimedia Tools and Applications,2022,81(7):9277-9287.
[78]ZHANG X,YE P,XIAO G.VIFB:a visible and infrared image fusion benchmark[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition Workshops.2020:104-105.
[79]LIU Z,BLASCH E,XUE Z,et al.Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision:A Comparative Study[J].IEEE Trans. Pattern. Anal. Mach. Intell,2012,34(1):94-109.
[80]ROBERTS J W,VAN AARDT J A,AHMED F B.Assessment of image fusion procedures using entropy,image quality,and multispectral classification[J].Journal of Applied Remote Sen-sing,2008,2(1):023522.
[81]QU G,ZHANG D,YAN P.Information measure for perfor-mance of image fusion[J].Electronics Letters,2002,38(7):313-315.
[82]CUI G,FENG H,XU Z,et al.Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J].Optics Communications,2015,341:199-209.
[83]XYDEAS C A,PETROVIC V.Objective image fusion perfor-mance measure[J].Electronics Letters,2000,36(4):308-309.
[84]ESKICIOGLU A M,FISHER P S.Image quality measures and their performance[J].IEEE Transactions on Communications,1995,43(12):2959-2965.
[85]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[1] 李帅, 徐彬, 韩祎珂, 廖同鑫.
SS-GCN:情感增强和句法增强的方面级情感分析模型
SS-GCN:Aspect-based Sentiment Analysis Model with Affective Enhancement and Syntactic Enhancement
计算机科学, 2023, 50(3): 3-11. https://doi.org/10.11896/jsjkx.220700238
[2] 陈富强, 寇嘉敏, 苏利敏, 李克.
基于图神经网络的多信息优化实体对齐模型
Multi-information Optimized Entity Alignment Model Based on Graph Neural Network
计算机科学, 2023, 50(3): 34-41. https://doi.org/10.11896/jsjkx.220700242
[3] 于健, 赵满坤, 高洁, 王聪源, 李亚蓉, 张文彬.
基于高阶和时序特征的图神经网络社会化推荐算法研究
Study on Graph Neural Networks Social Recommendation Based on High-order and Temporal Features
计算机科学, 2023, 50(3): 49-64. https://doi.org/10.11896/jsjkx.220700108
[4] 李志飞, 赵月, 张龑.
基于表示学习的知识图谱推理研究综述
Survey of Knowledge Graph Reasoning Based on Representation Learning
计算机科学, 2023, 50(3): 94-113. https://doi.org/10.11896/jsjkx.220900136
[5] 饶丹, 时宏伟.
基于深度聚类的航空交通流识别与异常检测研究
Study on Air Traffic Flow Recognition and Anomaly Detection Based on Deep Clustering
计算机科学, 2023, 50(3): 121-128. https://doi.org/10.11896/jsjkx.220100086
[6] 王晓飞, 樊学强, 李章维.
基于迁移学习和多视图特征融合提高RNA碱基相互作用预测
Improving RNA Base Interactions Prediction Based on Transfer Learning and Multi-view Feature Fusion
计算机科学, 2023, 50(3): 164-172. https://doi.org/10.11896/jsjkx.211200186
[7] 梅鹏程, 杨吉斌, 张强, 黄翔.
一种基于三维卷积的声学事件联合估计方法
Sound Event Joint Estimation Method Based on Three-dimension Convolution
计算机科学, 2023, 50(3): 191-198. https://doi.org/10.11896/jsjkx.220500259
[8] 胡中源, 薛羽, 查加杰.
演化循环神经网络研究综述
Survey on Evolutionary Recurrent Neural Networks
计算机科学, 2023, 50(3): 254-265. https://doi.org/10.11896/jsjkx.220600007
[9] 冯程程, 刘派, 姜琳颖, 梅笑寒, 郭贵冰.
文档增强型知识库问答
Document-enhanced Question Answering over Knowledge-Bases
计算机科学, 2023, 50(3): 266-275. https://doi.org/10.11896/jsjkx.220300022
[10] 环志刚, 蒋国权, 张玉健, 刘浏, 刘姗姗.
门控机制融合多种特征的中文事件共指消解
Employing Gated Mechanism to Incorporate Multi-features into Chinese Event Coreference Resolution
计算机科学, 2023, 50(3): 291-297. https://doi.org/10.11896/jsjkx.220700146
[11] 李海涛, 王瑞敏, 董卫宇, 蒋烈辉.
一种基于GRU的半监督网络流量异常检测方法
Semi-supervised Network Traffic Anomaly Detection Method Based on GRU
计算机科学, 2023, 50(3): 380-390. https://doi.org/10.11896/jsjkx.220100032
[12] 王祥炜, 韩锐, 刘驰.
基于层级化数据记忆池的边缘侧半监督持续学习方法
Hierarchical Memory Pool Based Edge Semi-supervised Continual Learning Method
计算机科学, 2023, 50(2): 23-31. https://doi.org/10.11896/jsjkx.221100133
[13] 章琪, 于双元, 尹鸿峰, 徐保民.
基于图注意力的神经协同过滤社会推荐算法
Neural Collaborative Filtering for Social Recommendation Algorithm Based on Graph Attention
计算机科学, 2023, 50(2): 115-122. https://doi.org/10.11896/jsjkx.211200019
[14] 曹金娟, 钱忠, 李培峰.
基于联合模型的端到端事件可信度识别
End-to-End Event Factuality Identification with Joint Model
计算机科学, 2023, 50(2): 292-299. https://doi.org/10.11896/jsjkx.211200108
[15] 郝敬宇, 文静轩, 刘华锋, 景丽萍, 于剑.
结合全局信息的深度图解耦协同过滤
Deep Disentangled Collaborative Filtering with Graph Global Information
计算机科学, 2023, 50(1): 41-51. https://doi.org/10.11896/jsjkx.220900255
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!