Computer Science ›› 2021, Vol. 48 ›› Issue (8): 185-190.doi: 10.11896/jsjkx.200600132

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Multi-band Image Self-supervised Fusion Method Based on Multi-discriminator

TIAN Song-wang, LIN Su-zhen, YANG Bo   

  1. College of Data Science and Technology,North University of China,Taiyuan 030051,China
  • Received:2020-06-20 Revised:2020-08-18 Published:2021-08-10
  • About author:TIAN Song-wang,born in 1996,postgraduate,is a member of China Computer Federation.His main research interests include image fusion and deep learning.( Su-zhen,born in 1966,Ph.D,professor,postgraduate supervisor,is a member of China Computer Federation.Her main research interests include image processing and information fusion.
  • Supported by:
    Applied Basic Research Project of Shanxi Province,China(201701D121062,201901D111151),Graduate Innovation Project of Shanxi Province,China(2020SY382) and 16th Graduate Science and Technology Project of North University of China(20191636).

Abstract: In order to solve the problem that the fusion result is limited due to the over dependence on the label image when using the deep learning methods in the multi band image fusion field,a multi-band image feature-level self-supervised fusion method based on multi-discriminator generation adversarial network is proposed.Firstly,this paper designs and builds a feedback dense network as a feature enhancement module to separately extract multi-band image features and perform feature enhancement.Se-condly,it merges and connects the multi-band image feature enhanced results and reconstructs the fused image through the designed feature fusion module.Finally,the preliminary fused result and the source images of each band are input into the discriminator network respectively.Through the classification task of multiple discriminators,the generator is continuously optimized so that the output of the generator retains the characteristics of multiple band images at the same time to achieve the purpose of image fusion.Experimental results show that,compared with the current representative fusion method,the proposed method has better clarity,information volume,more detailed information,and is more in line with human visual characteristics.

Key words: Deep learning, Dense network, Generative adversarial network, Image fusion, Multi-band image, Self-supervised learning

CLC Number: 

  • TP391
[1]MA J Y,MA Y,LI C.Infrared and visible image fusion methods and applications:A survey[J].Information Fusion,2019,45:153-178.
[2]MANVIYA M,BHARTI J.Image Fusion Survey:A Compre-hensive and Detailed Analysis of Image Fusion Techniques[M]//Social Networking and Computational Intelligence.Singapore:Springer,2020:649-659.
[3]LIN S Z,YANG X L,ZHAO J C.Multi-band image fusion based on intuitionistic fuzzy set sorting[C]//Proceedings of the 9th Chinese Information Fusion Conference.2019:808-813.
[4]SAIKIA P,BARUAH R D,SINGH S K,et al.Artificial Neural Networks in the domain of reservoir characterization:A review from shallow to deep models[J].Computers & Geosciences,2020,135(2):104357.
[5]MILLER D J,XIANG Z,KESIDIS G.Adversarial learning targeting deep neural network classification:a comprehensive review of defenses against attacks[J].Proceedings of the IEEE,2020,108(3):402-433.
[6]YANG X L,LIN S Z.A multi-band image feature level fusion method with attention mechanism[J].Journal of Xidian University,2020,47(1):120-127.
[7]LIN S Z,HAN Z,LI D W,et al.Integrating model-and data-driven methods for synchronous adaptive multi-band image fusion[J].Information Fusion,2020,54:145-160.
[8]JING L,TIAN Y.Self-supervised visual feature learning withdeep neural networks:A survey[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020(99):1-1.
[9]ZHAI X,OLIVER A,KOLESNIKOV A,et al.S4l:Self-super-vised semi-supervised learning[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:1476-1485.
[10]DAI Q,PATIL V,HECKER S,et al.Self-supervised ObjectMotion and Depth Estimation from Video[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2020:1004-1005.
[11]SHUKLA A,VOUGIOUKAS K,MA P,et al.Visually Guided Self Supervised Learning of Speech Representations[J].arXiv:2001.04316,2020.
[12]WANG Y D,ZHANG J,KAN M N,et al.Self-supervised Equi-variant Attention Mechanism for Weakly Supervised Semantic Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:12275-12284.
[13]GOODFELLOW I J,ABADIE J P,MIRZA M,et al.Generative adversarial nets[C]//Advances in Neural Information Proces-sing Systems.2014:2672-2680.
[14]MA J Y,YU W,LIANG P W,et al.FusionGAN:A generative adversarial network for infrared and visible image fusion[J].Information Fusion,2019,48:11-26.
[15]HUANG G,LIU Z,VAN D M L,et al.Densely connected con-volutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4700-4708.
[16]LI H,WU X J.Densefuse:A fusion approach to infrared and vi-sible images[J].IEEE Transactions on Image Processing,2018,28(5):2614-2623.
[17]LYU Q,SHAN H,STEBER C,et al.Multi-contrast super-resolution mri through a progressive network[J].IEEE Transactions on Medical Imaging,2020,39(9):2738-2749.
[18]GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improvedtraining of wasserstein gans[C]//Advances in Neural Information Processing Systems.2017:5767-5777.
[19]HAN Z.Research on multi-band image adaptive fusion method based on deep learning [D].Taiyuan:North University of China,2018.
[20]MOONON A U,HU J.Multi-focus image fusion based on NSCT and NSST[J].Sensing and Imaging,2015,16(1):4.
[1] ZHANG Jia, DONG Shou-bin. Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer [J]. Computer Science, 2022, 49(9): 41-47.
[2] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[3] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[4] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[5] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[6] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[7] HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[8] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[9] HU Yan-yu, ZHAO Long, DONG Xiang-jun. Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification [J]. Computer Science, 2022, 49(7): 73-78.
[10] DAI Zhao-xia, LI Jin-xin, ZHANG Xiang-dong, XU Xu, MEI Lin, ZHANG Liang. Super-resolution Reconstruction of MRI Based on DNGAN [J]. Computer Science, 2022, 49(7): 113-119.
[11] CHENG Cheng, JIANG Ai-lian. Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction [J]. Computer Science, 2022, 49(7): 120-126.
[12] HOU Yu-tao, ABULIZI Abudukelimu, ABUDUKELIMU Halidanmu. Advances in Chinese Pre-training Models [J]. Computer Science, 2022, 49(7): 148-163.
[13] ZHOU Hui, SHI Hao-chen, TU Yao-feng, HUANG Sheng-jun. Robust Deep Neural Network Learning Based on Active Sampling [J]. Computer Science, 2022, 49(7): 164-169.
[14] DU Hang-yuan, LI Duo, WANG Wen-jian. Method for Abnormal Users Detection Oriented to E-commerce Network [J]. Computer Science, 2022, 49(7): 170-178.
[15] SU Dan-ning, CAO Gui-tao, WANG Yan-nan, WANG Hong, REN He. Survey of Deep Learning for Radar Emitter Identification Based on Small Sample [J]. Computer Science, 2022, 49(7): 226-235.
Full text



No Suggested Reading articles found!