红外和可见光图像融合算法的研究进展

doi:10.11896/jsjkx.220100074

Abstract

Abstract: Infrared images are easy to identify thermal targets,and visible images have rich texture information.The fusion of infrared and visible images takes the advantages of both optical bands which can clearly show the targets and background.It has been widely used in many fields such as military reconnaissance,security monitoring,remote sensing measurement,and becomes a key research direction in the field of image fusion.In recent years,infrared and visible image fusion algorithms have attracted the attention of researchers around the world and have been studied abundantly.In this paper,the image fusion algorithms are introduced firstly,including traditional image processing methods of multi-scale transformation,sparse representation,and deep lear-ning algorithms based on CNN,GAN,AE.Then,the evaluation methods of fusion images are summarized,and a variety of common objective evaluation indexes are classified.After that,comparative experiments are carried out to subjectively evaluate and quantitatively analyze the advantages and disadvantages of these algorithms.Finally,the development trend of infrared and visible image fusion methods is prospected.

Key words: Image fusion, Infrared image, Visible image, Neural network, Evaluation metric

CLC Number:

TP391

WEI Qi, ZHAO Juan. Research Progress of Infrared and Visible Image Fusion Algorithms[J].Computer Science, 2023, 50(2): 190-200.

References

[1]LI S,KANG X,FANG L,et al.Pixel-level image fusion:A survey of the state of the art[J].information Fusion,2017,33:100-112.
[2]MA J,MA Y,LI C.Infrared and visible image fusion methods and applications:A survey[J].Information Fusion,2019,45:153-178.
[3]HOGERVORST M,TOET A.Fast natural color mapping fornight-time imagery[J].Information Fusion,2010,11(2):69-77.
[4]TOET A.Natural colour mapping for multiband nightvisionimagery[J].Information Fusion,2003,4(3):155-166.
[5]HAN J,BHANU B.Fusion of color and infrared video for mo-ving human detection[J].Pattern Recognition,2007,40(6):1771-1784.
[6]SINGH R,VATSA M,NOORE A.Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition[J].Pattern Recognition,2008,41(3):880-893.
[7]BHATNAGAR G,LIU Z.A novel image fusion framework for night-vision navigation and surveillance[J].Signal,Image and Video Processing,2015,9(1):165-175.
[8]PARAMANANDHAM N,RAJENDIRAN K.Multi sensorimage fusion for surveillance applications using hybrid image fusion algorithm[J].Multimedia Tools and Applications,2018,77(10):12405-12436.
[9]BULANON D,BURKS T,ALCHANATIS V.Image fusion ofvisible and thermal images for fruit detection[J].Biosystems engineering,2009,103(1):12-22.
[10]SANCHEZ V,PRINCE G,CLARKSON JP,et al.Registration of thermal and visible light images of diseased plants using silhouette extraction in the wavelet domain[J].Pattern Recognition,2015,48(7):2119-2128.
[11]SIMONE G,FARINA A,MORABITO F C,et al.Image fusiontechniques for remote sensing applications[J].Information fusion,2002,3(1):3-15.
[12]LI H,DING W,CAO X,et al.Image registration and fusion of visible and infrared integrated camera for medium-altitude unmanned aerial vehicle remote sensing[J].Remote Sensing,2017,9(5):441.
[13]BURT P J,ADELSON E H.The Laplacian Pyramid as a Compact Image Code[J].IEEE Transactions on Communications,1983,31(4):532-540.
[14]TOET A,VAN RUYVEN L J,VALETON J M.Merging thermal and visual images by a contrast pyramid[J].Optical Engineering,1989,28(7):287789.
[15]ZHANG Z,BLUM R S.A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application[C]//Proceedings of the IEEE.1999:1315-1326.
[16]CHEN J,LI X,LUO L,et al.Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J].Information Sciences,2020,508:64-78.
[17]LI S,KANG X,HU J.Image fusion with guided filtering[J].IEEE Trans Image Process,2013,22(7):2864-2875.
[18]ZHANG Q,LIU Y,BLUM R S,et al.Sparse representationbased multi-sensor image fusion for multi-focus and multi-modality images:A review[J].Information Fusion,2018,40:57-75.
[19]WANG K,QI G,ZHU Z,et al.A novel geometric dictionary construction approach for sparse representation based image fusion[J].Entropy,2017,19(7):306.
[20]YIN H.Sparse representation with learned multiscale dictionary for image fusion[J].Neurocomputing,2015,148:600-610.
[21]SUN C,ZHANG C,XIONG N.Infrared and Visible Image Fusion Techniques Based on Deep Learning:A Review[J].Electronics,2020,9(12):2162.
[22]MA J,YU W,LIANG P,et al.FusionGAN:A generative adversarial network for infrared and visible image fusion[J].Information Fusion,2019,48:11-26.
[23]PRABHAKAR K R,SRIKAR V S,BABU R V.Deepfuse:Adeep unsupervised approach for exposure fusion with extreme exposure image pairs[C]//Proceedings of the IEEE Interna-tional Conference on Computer Vision.2017:4714-4722.
[24]LIU Y,CHEN X,CHENG J,et al.Infrared and visible image fusion with convolutional neural networks[J].International Journal of Wavelets,Multiresolution and Information Processing,2018,16(3):1850018.
[25]LI H,WU X J,KITTLER J.RFN-Nest:An end-to-end residual fusion network for infrared and visible images[J].Information Fusion,2021,73:72-86.
[26]HE D X,MENG Y,WANG C Y.Contrast pyramid based imagefusion scheme for infrared image and visible image[C]//2011 IEEE International Geoscience and Remote Sensing Sympo-sium.2011:597-600.
[27]TOET A.A morphological pyramidal image decomposition[J].Pattern Recognition Letters,1989,9(4):255-261.
[28]GROSSMANN A,MORLET J.Decomposition of Hardy functions into square integrable wavelets of constant shape[J].SIAM Journal on Mathematical Analysis,1984,15(4):723-736.
[29]ZHAN L,ZHUANG Y,HUANG L.Infrared and visible images fusion method based on discrete wavelet transform[J].Journal of Computers,2017,28(2):57-71.
[30]HAN X,LI Z L,YAO D L,et al.Fusion of infrared and visible images based on discrete wavelet transform[C]//The Photoelec-tronic Technology Committee Conferences.2015:387-392.
[31]SELESNICK I W,BARANIUK R G,KINGSBURY N C.The dual-tree complex wavelet transform[J].IEEE Signal Processing Magazine,2005,22(6):123-151.
[32]ZUO Y,LIU J,BAI G,et al.Airborne infrared and visible image fusion combined with region segmentation[J].Sensors,2017,17(5):1127.
[33]MINGHUI S,LU L,YUANXI P,et al.Infrared & visible images fusion based on redundant directional lifting-based wavelet and saliency detection[J].Infrared Physics & Technology,2019,101:45-55.
[34]ZOU Y,LIANG X,WANG T.Visible and infrared image fusion using the lifting wavelet[J].TELKOMNIKA Indonesian Journal of Electrical Engineering,2013,11(11):6290-6295.
[35]DO M N,VETTERLI M.Contourlets:a directional multiresolution image representation[C]//Proceedings of International Conference on Image Processing.2002.
[36]DA CUNHA A L,ZHOU J,DO M N.The nonsubsampled con-tourlet transform:theory,design,and applications[J].IEEE Transactions on Image Processing,2006,15(10):3089-3101.
[37]MENG F,SONG M,GUO B,et al.Image fusion based on object region detection and Non-Subsampled Contourlet Transform[J].Computers & Electrical Engineering,2017,62:375-383.
[38]KONG W,WANG B,LEI Y.Technique for infrared and visible image fusion based on non-subsampled shearlet transform and spiking cortical model[J].Infrared Physics & Technology,2015,71:87-98.
[39]CAO Y,LI X,CHEN X,et al.Research on Fusion Method of Infrared and Visible Image Based on NSST and Guided Filtering[J].Journal of Changchun University of Science and Technology(Natural Science Edition),2022,45(3):28-34.
[40]LIU Y,LIU S,WANG Z.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion,2015,24:147-164.
[41]LIU G T,ZHOU J H,LI T,et al.Infrared and visible image fusion through hybrid curvature filtering image decomposition[J].Infrared Physics & Technology,2022,120:103938.
[42]YANG B,LI S.Multifocus image fusion and restoration withsparse representation[J].IEEE Transactions on Instrumentation and Measurement,2009,59(4):884-892.
[43]LIU C H,QI Y,DING W R.Infrared and visible image fusion method based on saliency detection in sparse domain[J].Infrared Physics & Technology,2017,83:94-102.
[44]BIN Y,CHAO Y,GUO Y H.Efficient image fusion with approximate sparse representation[J].International Journal of Wavelets,Multiresolution and Information Processing,2016,14(4):1650024.
[45]AHARON M,ELAD M,BRUCKSTEIN A.K-SVD:An algo-rithm for designing overcomplete dictionaries for sparse representation[J].IEEE Transactions on Signal Processing,2006,54(11):4311-4322.
[46]PATI Y C,REZAIIFAR R,KRISHNAPRASAD P S.Orthogonal matching pursuit:Recursive function approximation with applications to wavelet decomposition[C]//Proceedings of 27th Asilomar Conference on Signals,Systems and Computers.1993:40-44.
[47]YANG B,LI S.Pixel-level image fusion with simultaneous orthogonal matching pursuit[J].Information Fusion,2012,13(1):10-19.
[48]LI H,LIU L,HUANG W,et al.An improved fusion algorithm for infrared and visible images based on multi-scale transform[J].Infrared Physics & Technology,2016,74:28-37.
[49]MITIANOUDIS N,ANTONOPOULOS S A,STATHAKI T.Region-based ICA image fusion using textural information[C]//2013 18th International Conference on Digital Signal Processing(DSP).2013:1-6.
[50]KONG W,LEI Y,ZHAO H.Adaptive fusion method of visible light and infrared images based on non-subsampled shearlet transform and fast non-negative matrix factorization[J].Infrared Physics & Technology,2014,67:161-172.
[51]CAI J J,CHENG Q M,PENG M J,et al.Fusion of infrared and visible images based on nonsubsampled contourlet transform and sparse K-SVD dictionary learning[J].Infrared Physics & Technology,2017,82:85-95.
[52]CHEN G,CHEN Y,LI J,et al.Infrared and visible image fusionbased on discrete nonseparable shearlet transform and convolutional sparse representation[J].Journal of Jilin University(Engineering and Technology Edition),2021,51(3):996-1010.
[53]LIU Y,LIU S P,WANG Z F.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion,2015,24:147-164.
[54]ZHANG X Y,MA Y,FAN F,et al.Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition[J].Journal of the Optical Society of America A－Optics Image Science and Vision,2017,34(8):1400-1410.
[55]HOU R C,ZHOU D M,NIE R C,et al.Infrared and VisibleImages Fusion Using Visual Saliency and Dual-PCNN[J].Com-puter Science,2018,45(6A):162-166.
[56]LIU Y,CHEN X,WARD R K,et al.Image fusion with convolutional sparse representation[J].IEEE Signal Processing Letters,2016,23(12):1882-1886.
[57]LIU Y,CHEN X,PENG H,et al.Multi-focus image fusion with a deep convolutional neural network[J].Information Fusion,2017,36:191-207.
[58]LI H,WU X J,KITTLER J.Infrared and visible image fusionusing a deep learning framework[C]//2018 24th International Conference on Pattern Recognition(ICPR).2018:2705-2710.
[59]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[60]LI H,WU X J,DURRANI T S.Infrared and visible image fusion with ResNet and zero-phase component analysis[J].Infrared Physics & Technology,2019,102:103039.
[61]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[62]KESSY A,LEWIN A,STRIMMER K.Optimal whitening anddecorrelation[J].The American Statistician,2018,72(4):309-314.
[63]ZHANG Y,LIU Y,SUN P,et al.IFCNN:A general image fusion framework based on convolutional neural network[J].Information Fusion,2020,54:99-118.
[64]XU H,MA J,LE Z,et al.Fusiondn:A unified densely connected network for image fusion[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:12484-12491.
[65]HUANG G,LIU Z,VAN DER MAATEN L,et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4700-4708.
[66]XU H,MA J,JIANG J,et al.U2Fusion:A Unified Unsuper-vised Image Fusion Network[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,44(1):502-518.
[67]LI C,HOU J,LI J,et al.Infrared and Visible Image Fusion Method Based on Attention and Residual Concatenation[J].Computer Engineering,2022,48(7):234-240.
[68]MA J,LIANG P,YU W,et al.Infrared and visible image fusion via detail preserving adversarial learning[J].Information Fusion,2020,54:85-98.
[69]YUAN C,SUN C,TANG X,et al.FLGC-Fusion GAN:An Enhanced Fusion GAN Model by Importing Fully Learnable Group Convolution[J].Mathematical Problems in Engineering,2020,2020:6384831.
[70]LI Q,LU L,LI Z,et al.Coupled GAN With Relativistic Discri-minators for Infrared and Visible Images Fusion[J].IEEE Sensors Journal,2021,21(6):7458-7467.
[71]TANG L L,LIU G,XIAO G,et al.p Infrared and visible image fusion based on guided hybrid model and generative adversarial network[J].Infrared Physics & Technology,2022,120:103914.
[72]LI H,WU X J.DenseFuse:A fusion approach to infrared andvisible images[J].IEEE Transactions on Image Processing,2018,28(5):2614-2623.
[73]LI H,WU X J,DURRANI T.NestFuse:An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models[J].IEEE Transactions on Instrumentation and Measurement,2020,69(12):9645-9656.
[74]ZHOU Z,SIDDIQUEE M M R,TAJBAKHSH N,et al.UNet++:A Nested U-Net Architecture for Medical Image Segmentation[C]//Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support.2018:3-11.
[75]FU Y,WU X J.A dual-branch network for infrared and visible image fusion[C]//2020 25th International Conference on Pattern Recognition(ICPR).2021:10675-10680.
[76]XU H,ZHANG H,MA J.Classification Saliency-Based Rulefor Visible and Infrared Image Fusion[J].IEEE Transactions on Computational Imaging,2021,7:824-836.
[77]ZHANG L,LI H,ZHU R,et al.An infrared and visible image fusion algorithm based on ResNet-152[J].Multimedia Tools and Applications,2022,81(7):9277-9287.
[78]ZHANG X,YE P,XIAO G.VIFB:a visible and infrared image fusion benchmark[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition Workshops.2020:104-105.
[79]LIU Z,BLASCH E,XUE Z,et al.Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision:A Comparative Study[J].IEEE Trans. Pattern. Anal. Mach. Intell,2012,34(1):94-109.
[80]ROBERTS J W,VAN AARDT J A,AHMED F B.Assessment of image fusion procedures using entropy,image quality,and multispectral classification[J].Journal of Applied Remote Sen-sing,2008,2(1):023522.
[81]QU G,ZHANG D,YAN P.Information measure for perfor-mance of image fusion[J].Electronics Letters,2002,38(7):313-315.
[82]CUI G,FENG H,XU Z,et al.Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J].Optics Communications,2015,341:199-209.
[83]XYDEAS C A,PETROVIC V.Objective image fusion perfor-mance measure[J].Electronics Letters,2000,36(4):308-309.
[84]ESKICIOGLU A M,FISHER P S.Image quality measures and their performance[J].IEEE Transactions on Communications,1995,43(12):2959-2965.
[85]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Research Progress of Infrared and Visible Image Fusion Algorithms

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 15

Metrics

Comments

Recommended 0

[1]	CHEN Fuqiang, KOU Jiamin, SU Limin, LI Ke. Multi-information Optimized Entity Alignment Model Based on Graph Neural Network [J]. Computer Science, 2023, 50(3): 34-41.
[2]	YU Jian, ZHAO Mankun, GAO Jie, WANG Congyuan, LI Yarong, ZHANG Wenbin. Study on Graph Neural Networks Social Recommendation Based on High-order and Temporal Features [J]. Computer Science, 2023, 50(3): 49-64.
[3]	LI Zhifei, ZHAO Yue, ZHANG Yan. Survey of Knowledge Graph Reasoning Based on Representation Learning [J]. Computer Science, 2023, 50(3): 94-113.
[4]	RAO Dan, SHI Hongwei. Study on Air Traffic Flow Recognition and Anomaly Detection Based on Deep Clustering [J]. Computer Science, 2023, 50(3): 121-128.
[5]	WANG Xiaofei, FAN Xueqiang, LI Zhangwei. Improving RNA Base Interactions Prediction Based on Transfer Learning and Multi-view Feature Fusion [J]. Computer Science, 2023, 50(3): 164-172.
[6]	MEI Pengcheng, YANG Jibin, ZHANG Qiang, HUANG Xiang. Sound Event Joint Estimation Method Based on Three-dimension Convolution [J]. Computer Science, 2023, 50(3): 191-198.
[7]	HU Zhongyuan, XUE Yu, ZHA Jiajie. Survey on Evolutionary Recurrent Neural Networks [J]. Computer Science, 2023, 50(3): 254-265.
[8]	FENG Chengcheng, LIU Pai, JIANG Linying, MEI Xiaohan, GUO Guibing. Document-enhanced Question Answering over Knowledge-Bases [J]. Computer Science, 2023, 50(3): 266-275.
[9]	HUAN Zhigang, JIANG Guoquan, ZHANG Yujian, LIU Liu, LIU Shanshan. Employing Gated Mechanism to Incorporate Multi-features into Chinese Event Coreference Resolution [J]. Computer Science, 2023, 50(3): 291-297.
[10]	LI Haitao, WANG Ruimin, DONG Weiyu, JIANG Liehui. Semi-supervised Network Traffic Anomaly Detection Method Based on GRU [J]. Computer Science, 2023, 50(3): 380-390.
[11]	ZHANG Qi, YU Shuangyuan, YIN Hongfeng, XU Baomin. Neural Collaborative Filtering for Social Recommendation Algorithm Based on Graph Attention [J]. Computer Science, 2023, 50(2): 115-122.
[12]	WANG Xiangwei, HAN Rui, Chi Harold LIU. Hierarchical Memory Pool Based Edge Semi-supervised Continual Learning Method [J]. Computer Science, 2023, 50(2): 23-31.
[13]	PU Jinyao, BU Lingmei, LU Yongmei, YE Ziming, CHEN Li, YU Zhonghua. Utilizing Heterogeneous Graph Neural Network to Extract Emotion-Cause Pairs Effectively [J]. Computer Science, 2023, 50(1): 205-212.
[14]	ZHANG Rujia, DAI Lu, GUO Peng, WANG Bang. Chinese Nested Named Entity Recognition Algorithm Based on Segmentation Attention andBoundary-aware [J]. Computer Science, 2023, 50(1): 213-220.
[15]	HUANG Zenan, LIU Xiaojie, ZHAO Chenhui, DENG Yabin, GUO Donghui. Spiking Neural Network Model for Brain-like Computing and Progress of Its Learning Algorithm [J]. Computer Science, 2023, 50(1): 229-242.