计算机科学 ›› 2023, Vol. 50 ›› Issue (2): 190-200.doi: 10.11896/jsjkx.220100074
魏琦1,2, 赵娟1
WEI Qi1,2, ZHAO Juan1
摘要: 红外图像便于识别热源目标,可见光图像包含丰富的纹理信息。红外和可见光的融合图像兼顾了两个波段传感器的优势,可以清楚地显示热源目标及其背景,在军事侦察、安防监控、遥感监测等领域有着广泛的应用,已成为图像融合领域的重点研究方向。近年来,国内外学者对红外和可见光图像融合算法开展了大量研究。文中首先对现有的图像融合算法进行了详细介绍,包括多尺度变换、稀疏表示的传统图像处理方法和基于CNN,GAN,AE这3种常见网络结构的深度学习图像融合算法。接着综述了融合图像的评价方法,对常见的多种客观评价指标进行了归类分析。然后开展对比实验,对各种方法进行了主观评价和定量分析,指出不同方法的优势和不足。最后,对红外和可见光图像融合技术的未来发展趋势进行展望。
中图分类号:
[1]LI S,KANG X,FANG L,et al.Pixel-level image fusion:A survey of the state of the art[J].information Fusion,2017,33:100-112. [2]MA J,MA Y,LI C.Infrared and visible image fusion methods and applications:A survey[J].Information Fusion,2019,45:153-178. [3]HOGERVORST M,TOET A.Fast natural color mapping fornight-time imagery[J].Information Fusion,2010,11(2):69-77. [4]TOET A.Natural colour mapping for multiband nightvisionimagery[J].Information Fusion,2003,4(3):155-166. [5]HAN J,BHANU B.Fusion of color and infrared video for mo-ving human detection[J].Pattern Recognition,2007,40(6):1771-1784. [6]SINGH R,VATSA M,NOORE A.Integrated multilevel image fusion and match score fusion of visible and infrared face images for robust face recognition[J].Pattern Recognition,2008,41(3):880-893. [7]BHATNAGAR G,LIU Z.A novel image fusion framework for night-vision navigation and surveillance[J].Signal,Image and Video Processing,2015,9(1):165-175. [8]PARAMANANDHAM N,RAJENDIRAN K.Multi sensorimage fusion for surveillance applications using hybrid image fusion algorithm[J].Multimedia Tools and Applications,2018,77(10):12405-12436. [9]BULANON D,BURKS T,ALCHANATIS V.Image fusion ofvisible and thermal images for fruit detection[J].Biosystems engineering,2009,103(1):12-22. [10]SANCHEZ V,PRINCE G,CLARKSON JP,et al.Registration of thermal and visible light images of diseased plants using silhouette extraction in the wavelet domain[J].Pattern Recognition,2015,48(7):2119-2128. [11]SIMONE G,FARINA A,MORABITO F C,et al.Image fusiontechniques for remote sensing applications[J].Information fusion,2002,3(1):3-15. [12]LI H,DING W,CAO X,et al.Image registration and fusion of visible and infrared integrated camera for medium-altitude unmanned aerial vehicle remote sensing[J].Remote Sensing,2017,9(5):441. [13]BURT P J,ADELSON E H.The Laplacian Pyramid as a Compact Image Code[J].IEEE Transactions on Communications,1983,31(4):532-540. [14]TOET A,VAN RUYVEN L J,VALETON J M.Merging thermal and visual images by a contrast pyramid[J].Optical Engineering,1989,28(7):287789. [15]ZHANG Z,BLUM R S.A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application[C]//Proceedings of the IEEE.1999:1315-1326. [16]CHEN J,LI X,LUO L,et al.Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J].Information Sciences,2020,508:64-78. [17]LI S,KANG X,HU J.Image fusion with guided filtering[J].IEEE Trans Image Process,2013,22(7):2864-2875. [18]ZHANG Q,LIU Y,BLUM R S,et al.Sparse representationbased multi-sensor image fusion for multi-focus and multi-modality images:A review[J].Information Fusion,2018,40:57-75. [19]WANG K,QI G,ZHU Z,et al.A novel geometric dictionary construction approach for sparse representation based image fusion[J].Entropy,2017,19(7):306. [20]YIN H.Sparse representation with learned multiscale dictionary for image fusion[J].Neurocomputing,2015,148:600-610. [21]SUN C,ZHANG C,XIONG N.Infrared and Visible Image Fusion Techniques Based on Deep Learning:A Review[J].Electronics,2020,9(12):2162. [22]MA J,YU W,LIANG P,et al.FusionGAN:A generative adversarial network for infrared and visible image fusion[J].Information Fusion,2019,48:11-26. [23]PRABHAKAR K R,SRIKAR V S,BABU R V.Deepfuse:Adeep unsupervised approach for exposure fusion with extreme exposure image pairs[C]//Proceedings of the IEEE Interna-tional Conference on Computer Vision.2017:4714-4722. [24]LIU Y,CHEN X,CHENG J,et al.Infrared and visible image fusion with convolutional neural networks[J].International Journal of Wavelets,Multiresolution and Information Processing,2018,16(3):1850018. [25]LI H,WU X J,KITTLER J.RFN-Nest:An end-to-end residual fusion network for infrared and visible images[J].Information Fusion,2021,73:72-86. [26]HE D X,MENG Y,WANG C Y.Contrast pyramid based imagefusion scheme for infrared image and visible image[C]//2011 IEEE International Geoscience and Remote Sensing Sympo-sium.2011:597-600. [27]TOET A.A morphological pyramidal image decomposition[J].Pattern Recognition Letters,1989,9(4):255-261. [28]GROSSMANN A,MORLET J.Decomposition of Hardy functions into square integrable wavelets of constant shape[J].SIAM Journal on Mathematical Analysis,1984,15(4):723-736. [29]ZHAN L,ZHUANG Y,HUANG L.Infrared and visible images fusion method based on discrete wavelet transform[J].Journal of Computers,2017,28(2):57-71. [30]HAN X,LI Z L,YAO D L,et al.Fusion of infrared and visible images based on discrete wavelet transform[C]//The Photoelec-tronic Technology Committee Conferences.2015:387-392. [31]SELESNICK I W,BARANIUK R G,KINGSBURY N C.The dual-tree complex wavelet transform[J].IEEE Signal Processing Magazine,2005,22(6):123-151. [32]ZUO Y,LIU J,BAI G,et al.Airborne infrared and visible image fusion combined with region segmentation[J].Sensors,2017,17(5):1127. [33]MINGHUI S,LU L,YUANXI P,et al.Infrared & visible images fusion based on redundant directional lifting-based wavelet and saliency detection[J].Infrared Physics & Technology,2019,101:45-55. [34]ZOU Y,LIANG X,WANG T.Visible and infrared image fusion using the lifting wavelet[J].TELKOMNIKA Indonesian Journal of Electrical Engineering,2013,11(11):6290-6295. [35]DO M N,VETTERLI M.Contourlets:a directional multiresolution image representation[C]//Proceedings of International Conference on Image Processing.2002. [36]DA CUNHA A L,ZHOU J,DO M N.The nonsubsampled con-tourlet transform:theory,design,and applications[J].IEEE Transactions on Image Processing,2006,15(10):3089-3101. [37]MENG F,SONG M,GUO B,et al.Image fusion based on object region detection and Non-Subsampled Contourlet Transform[J].Computers & Electrical Engineering,2017,62:375-383. [38]KONG W,WANG B,LEI Y.Technique for infrared and visible image fusion based on non-subsampled shearlet transform and spiking cortical model[J].Infrared Physics & Technology,2015,71:87-98. [39]CAO Y,LI X,CHEN X,et al.Research on Fusion Method of Infrared and Visible Image Based on NSST and Guided Filtering[J].Journal of Changchun University of Science and Technology(Natural Science Edition),2022,45(3):28-34. [40]LIU Y,LIU S,WANG Z.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion,2015,24:147-164. [41]LIU G T,ZHOU J H,LI T,et al.Infrared and visible image fusion through hybrid curvature filtering image decomposition[J].Infrared Physics & Technology,2022,120:103938. [42]YANG B,LI S.Multifocus image fusion and restoration withsparse representation[J].IEEE Transactions on Instrumentation and Measurement,2009,59(4):884-892. [43]LIU C H,QI Y,DING W R.Infrared and visible image fusion method based on saliency detection in sparse domain[J].Infrared Physics & Technology,2017,83:94-102. [44]BIN Y,CHAO Y,GUO Y H.Efficient image fusion with approximate sparse representation[J].International Journal of Wavelets,Multiresolution and Information Processing,2016,14(4):1650024. [45]AHARON M,ELAD M,BRUCKSTEIN A.K-SVD:An algo-rithm for designing overcomplete dictionaries for sparse representation[J].IEEE Transactions on Signal Processing,2006,54(11):4311-4322. [46]PATI Y C,REZAIIFAR R,KRISHNAPRASAD P S.Orthogonal matching pursuit:Recursive function approximation with applications to wavelet decomposition[C]//Proceedings of 27th Asilomar Conference on Signals,Systems and Computers.1993:40-44. [47]YANG B,LI S.Pixel-level image fusion with simultaneous orthogonal matching pursuit[J].Information Fusion,2012,13(1):10-19. [48]LI H,LIU L,HUANG W,et al.An improved fusion algorithm for infrared and visible images based on multi-scale transform[J].Infrared Physics & Technology,2016,74:28-37. [49]MITIANOUDIS N,ANTONOPOULOS S A,STATHAKI T.Region-based ICA image fusion using textural information[C]//2013 18th International Conference on Digital Signal Processing(DSP).2013:1-6. [50]KONG W,LEI Y,ZHAO H.Adaptive fusion method of visible light and infrared images based on non-subsampled shearlet transform and fast non-negative matrix factorization[J].Infrared Physics & Technology,2014,67:161-172. [51]CAI J J,CHENG Q M,PENG M J,et al.Fusion of infrared and visible images based on nonsubsampled contourlet transform and sparse K-SVD dictionary learning[J].Infrared Physics & Technology,2017,82:85-95. [52]CHEN G,CHEN Y,LI J,et al.Infrared and visible image fusionbased on discrete nonseparable shearlet transform and convolutional sparse representation[J].Journal of Jilin University(Engineering and Technology Edition),2021,51(3):996-1010. [53]LIU Y,LIU S P,WANG Z F.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion,2015,24:147-164. [54]ZHANG X Y,MA Y,FAN F,et al.Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition[J].Journal of the Optical Society of America A-Optics Image Science and Vision,2017,34(8):1400-1410. [55]HOU R C,ZHOU D M,NIE R C,et al.Infrared and VisibleImages Fusion Using Visual Saliency and Dual-PCNN[J].Com-puter Science,2018,45(6A):162-166. [56]LIU Y,CHEN X,WARD R K,et al.Image fusion with convolutional sparse representation[J].IEEE Signal Processing Letters,2016,23(12):1882-1886. [57]LIU Y,CHEN X,PENG H,et al.Multi-focus image fusion with a deep convolutional neural network[J].Information Fusion,2017,36:191-207. [58]LI H,WU X J,KITTLER J.Infrared and visible image fusionusing a deep learning framework[C]//2018 24th International Conference on Pattern Recognition(ICPR).2018:2705-2710. [59]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014. [60]LI H,WU X J,DURRANI T S.Infrared and visible image fusion with ResNet and zero-phase component analysis[J].Infrared Physics & Technology,2019,102:103039. [61]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778. [62]KESSY A,LEWIN A,STRIMMER K.Optimal whitening anddecorrelation[J].The American Statistician,2018,72(4):309-314. [63]ZHANG Y,LIU Y,SUN P,et al.IFCNN:A general image fusion framework based on convolutional neural network[J].Information Fusion,2020,54:99-118. [64]XU H,MA J,LE Z,et al.Fusiondn:A unified densely connected network for image fusion[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:12484-12491. [65]HUANG G,LIU Z,VAN DER MAATEN L,et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4700-4708. [66]XU H,MA J,JIANG J,et al.U2Fusion:A Unified Unsuper-vised Image Fusion Network[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,44(1):502-518. [67]LI C,HOU J,LI J,et al.Infrared and Visible Image Fusion Method Based on Attention and Residual Concatenation[J].Computer Engineering,2022,48(7):234-240. [68]MA J,LIANG P,YU W,et al.Infrared and visible image fusion via detail preserving adversarial learning[J].Information Fusion,2020,54:85-98. [69]YUAN C,SUN C,TANG X,et al.FLGC-Fusion GAN:An Enhanced Fusion GAN Model by Importing Fully Learnable Group Convolution[J].Mathematical Problems in Engineering,2020,2020:6384831. [70]LI Q,LU L,LI Z,et al.Coupled GAN With Relativistic Discri-minators for Infrared and Visible Images Fusion[J].IEEE Sensors Journal,2021,21(6):7458-7467. [71]TANG L L,LIU G,XIAO G,et al.p Infrared and visible image fusion based on guided hybrid model and generative adversarial network[J].Infrared Physics & Technology,2022,120:103914. [72]LI H,WU X J.DenseFuse:A fusion approach to infrared andvisible images[J].IEEE Transactions on Image Processing,2018,28(5):2614-2623. [73]LI H,WU X J,DURRANI T.NestFuse:An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models[J].IEEE Transactions on Instrumentation and Measurement,2020,69(12):9645-9656. [74]ZHOU Z,SIDDIQUEE M M R,TAJBAKHSH N,et al.UNet++:A Nested U-Net Architecture for Medical Image Segmentation[C]//Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support.2018:3-11. [75]FU Y,WU X J.A dual-branch network for infrared and visible image fusion[C]//2020 25th International Conference on Pattern Recognition(ICPR).2021:10675-10680. [76]XU H,ZHANG H,MA J.Classification Saliency-Based Rulefor Visible and Infrared Image Fusion[J].IEEE Transactions on Computational Imaging,2021,7:824-836. [77]ZHANG L,LI H,ZHU R,et al.An infrared and visible image fusion algorithm based on ResNet-152[J].Multimedia Tools and Applications,2022,81(7):9277-9287. [78]ZHANG X,YE P,XIAO G.VIFB:a visible and infrared image fusion benchmark[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition Workshops.2020:104-105. [79]LIU Z,BLASCH E,XUE Z,et al.Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision:A Comparative Study[J].IEEE Trans. Pattern. Anal. Mach. Intell,2012,34(1):94-109. [80]ROBERTS J W,VAN AARDT J A,AHMED F B.Assessment of image fusion procedures using entropy,image quality,and multispectral classification[J].Journal of Applied Remote Sen-sing,2008,2(1):023522. [81]QU G,ZHANG D,YAN P.Information measure for perfor-mance of image fusion[J].Electronics Letters,2002,38(7):313-315. [82]CUI G,FENG H,XU Z,et al.Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J].Optics Communications,2015,341:199-209. [83]XYDEAS C A,PETROVIC V.Objective image fusion perfor-mance measure[J].Electronics Letters,2000,36(4):308-309. [84]ESKICIOGLU A M,FISHER P S.Image quality measures and their performance[J].IEEE Transactions on Communications,1995,43(12):2959-2965. [85]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612. |
[1] | 李帅, 徐彬, 韩祎珂, 廖同鑫. SS-GCN:情感增强和句法增强的方面级情感分析模型 SS-GCN:Aspect-based Sentiment Analysis Model with Affective Enhancement and Syntactic Enhancement 计算机科学, 2023, 50(3): 3-11. https://doi.org/10.11896/jsjkx.220700238 |
[2] | 陈富强, 寇嘉敏, 苏利敏, 李克. 基于图神经网络的多信息优化实体对齐模型 Multi-information Optimized Entity Alignment Model Based on Graph Neural Network 计算机科学, 2023, 50(3): 34-41. https://doi.org/10.11896/jsjkx.220700242 |
[3] | 于健, 赵满坤, 高洁, 王聪源, 李亚蓉, 张文彬. 基于高阶和时序特征的图神经网络社会化推荐算法研究 Study on Graph Neural Networks Social Recommendation Based on High-order and Temporal Features 计算机科学, 2023, 50(3): 49-64. https://doi.org/10.11896/jsjkx.220700108 |
[4] | 李志飞, 赵月, 张龑. 基于表示学习的知识图谱推理研究综述 Survey of Knowledge Graph Reasoning Based on Representation Learning 计算机科学, 2023, 50(3): 94-113. https://doi.org/10.11896/jsjkx.220900136 |
[5] | 饶丹, 时宏伟. 基于深度聚类的航空交通流识别与异常检测研究 Study on Air Traffic Flow Recognition and Anomaly Detection Based on Deep Clustering 计算机科学, 2023, 50(3): 121-128. https://doi.org/10.11896/jsjkx.220100086 |
[6] | 王晓飞, 樊学强, 李章维. 基于迁移学习和多视图特征融合提高RNA碱基相互作用预测 Improving RNA Base Interactions Prediction Based on Transfer Learning and Multi-view Feature Fusion 计算机科学, 2023, 50(3): 164-172. https://doi.org/10.11896/jsjkx.211200186 |
[7] | 梅鹏程, 杨吉斌, 张强, 黄翔. 一种基于三维卷积的声学事件联合估计方法 Sound Event Joint Estimation Method Based on Three-dimension Convolution 计算机科学, 2023, 50(3): 191-198. https://doi.org/10.11896/jsjkx.220500259 |
[8] | 胡中源, 薛羽, 查加杰. 演化循环神经网络研究综述 Survey on Evolutionary Recurrent Neural Networks 计算机科学, 2023, 50(3): 254-265. https://doi.org/10.11896/jsjkx.220600007 |
[9] | 冯程程, 刘派, 姜琳颖, 梅笑寒, 郭贵冰. 文档增强型知识库问答 Document-enhanced Question Answering over Knowledge-Bases 计算机科学, 2023, 50(3): 266-275. https://doi.org/10.11896/jsjkx.220300022 |
[10] | 环志刚, 蒋国权, 张玉健, 刘浏, 刘姗姗. 门控机制融合多种特征的中文事件共指消解 Employing Gated Mechanism to Incorporate Multi-features into Chinese Event Coreference Resolution 计算机科学, 2023, 50(3): 291-297. https://doi.org/10.11896/jsjkx.220700146 |
[11] | 李海涛, 王瑞敏, 董卫宇, 蒋烈辉. 一种基于GRU的半监督网络流量异常检测方法 Semi-supervised Network Traffic Anomaly Detection Method Based on GRU 计算机科学, 2023, 50(3): 380-390. https://doi.org/10.11896/jsjkx.220100032 |
[12] | 王祥炜, 韩锐, 刘驰. 基于层级化数据记忆池的边缘侧半监督持续学习方法 Hierarchical Memory Pool Based Edge Semi-supervised Continual Learning Method 计算机科学, 2023, 50(2): 23-31. https://doi.org/10.11896/jsjkx.221100133 |
[13] | 章琪, 于双元, 尹鸿峰, 徐保民. 基于图注意力的神经协同过滤社会推荐算法 Neural Collaborative Filtering for Social Recommendation Algorithm Based on Graph Attention 计算机科学, 2023, 50(2): 115-122. https://doi.org/10.11896/jsjkx.211200019 |
[14] | 曹金娟, 钱忠, 李培峰. 基于联合模型的端到端事件可信度识别 End-to-End Event Factuality Identification with Joint Model 计算机科学, 2023, 50(2): 292-299. https://doi.org/10.11896/jsjkx.211200108 |
[15] | 郝敬宇, 文静轩, 刘华锋, 景丽萍, 于剑. 结合全局信息的深度图解耦协同过滤 Deep Disentangled Collaborative Filtering with Graph Global Information 计算机科学, 2023, 50(1): 41-51. https://doi.org/10.11896/jsjkx.220900255 |
|