计算机科学 ›› 2021, Vol. 48 ›› Issue (8): 185-190.doi: 10.11896/jsjkx.200600132

• 计算机图形学& 多媒体 • 上一篇    下一篇

基于多判别器的多波段图像自监督融合方法

田嵩旺, 蔺素珍, 杨博   

  1. 中北大学大数据学院 太原030051
  • 收稿日期:2020-06-20 修回日期:2020-08-18 发布日期:2021-08-10
  • 通讯作者: 蔺素珍(lsz@nuc.edu.cn)
  • 基金资助:
    山西省应用基础研究项目(201701D121062,201901D111151);山西省研究生创新项目(2020SY382);中北大学第十六届研究生科技立项(20191636)

Multi-band Image Self-supervised Fusion Method Based on Multi-discriminator

TIAN Song-wang, LIN Su-zhen, YANG Bo   

  1. College of Data Science and Technology,North University of China,Taiyuan 030051,China
  • Received:2020-06-20 Revised:2020-08-18 Published:2021-08-10
  • About author:TIAN Song-wang,born in 1996,postgraduate,is a member of China Computer Federation.His main research interests include image fusion and deep learning.(1092502682@qq.com)LIN Su-zhen,born in 1966,Ph.D,professor,postgraduate supervisor,is a member of China Computer Federation.Her main research interests include image processing and information fusion.
  • Supported by:
    Applied Basic Research Project of Shanxi Province,China(201701D121062,201901D111151),Graduate Innovation Project of Shanxi Province,China(2020SY382) and 16th Graduate Science and Technology Project of North University of China(20191636).

摘要: 针对多波段图像融合领域利用深度学习方法进行融合时过度依赖标签图像导致融合结果受限的问题,文中提出了一种基于多判别器生成对抗网络的多波段图像自监督融合方法。首先,设计并构建反馈密集网络作为特征增强模块,分别提取多波段图像特征并进行特征增强;其次,将多波段图像特征增强结果合并连接,并通过设计的特征融合模块重构融合图像;最后,将初步融合结果与各波段源图像分别输入判别网络,通过多个判别器的分类任务来不断优化生成器,使生成器在输出最终结果的同时保留多个波段图像的特征,以达到图像融合的目的。实验结果表明,与当前代表性的融合方法相比,所提方法具有更好的清晰度和更多信息量,细节信息更丰富,更符合人眼的视觉特性。

关键词: 多波段图像, 密集网络, 深度学习, 生成对抗网络, 图像融合, 自监督学习

Abstract: In order to solve the problem that the fusion result is limited due to the over dependence on the label image when using the deep learning methods in the multi band image fusion field,a multi-band image feature-level self-supervised fusion method based on multi-discriminator generation adversarial network is proposed.Firstly,this paper designs and builds a feedback dense network as a feature enhancement module to separately extract multi-band image features and perform feature enhancement.Se-condly,it merges and connects the multi-band image feature enhanced results and reconstructs the fused image through the designed feature fusion module.Finally,the preliminary fused result and the source images of each band are input into the discriminator network respectively.Through the classification task of multiple discriminators,the generator is continuously optimized so that the output of the generator retains the characteristics of multiple band images at the same time to achieve the purpose of image fusion.Experimental results show that,compared with the current representative fusion method,the proposed method has better clarity,information volume,more detailed information,and is more in line with human visual characteristics.

Key words: Deep learning, Dense network, Generative adversarial network, Image fusion, Multi-band image, Self-supervised learning

中图分类号: 

  • TP391
[1]MA J Y,MA Y,LI C.Infrared and visible image fusion methods and applications:A survey[J].Information Fusion,2019,45:153-178.
[2]MANVIYA M,BHARTI J.Image Fusion Survey:A Compre-hensive and Detailed Analysis of Image Fusion Techniques[M]//Social Networking and Computational Intelligence.Singapore:Springer,2020:649-659.
[3]LIN S Z,YANG X L,ZHAO J C.Multi-band image fusion based on intuitionistic fuzzy set sorting[C]//Proceedings of the 9th Chinese Information Fusion Conference.2019:808-813.
[4]SAIKIA P,BARUAH R D,SINGH S K,et al.Artificial Neural Networks in the domain of reservoir characterization:A review from shallow to deep models[J].Computers & Geosciences,2020,135(2):104357.
[5]MILLER D J,XIANG Z,KESIDIS G.Adversarial learning targeting deep neural network classification:a comprehensive review of defenses against attacks[J].Proceedings of the IEEE,2020,108(3):402-433.
[6]YANG X L,LIN S Z.A multi-band image feature level fusion method with attention mechanism[J].Journal of Xidian University,2020,47(1):120-127.
[7]LIN S Z,HAN Z,LI D W,et al.Integrating model-and data-driven methods for synchronous adaptive multi-band image fusion[J].Information Fusion,2020,54:145-160.
[8]JING L,TIAN Y.Self-supervised visual feature learning withdeep neural networks:A survey[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020(99):1-1.
[9]ZHAI X,OLIVER A,KOLESNIKOV A,et al.S4l:Self-super-vised semi-supervised learning[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:1476-1485.
[10]DAI Q,PATIL V,HECKER S,et al.Self-supervised ObjectMotion and Depth Estimation from Video[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2020:1004-1005.
[11]SHUKLA A,VOUGIOUKAS K,MA P,et al.Visually Guided Self Supervised Learning of Speech Representations[J].arXiv:2001.04316,2020.
[12]WANG Y D,ZHANG J,KAN M N,et al.Self-supervised Equi-variant Attention Mechanism for Weakly Supervised Semantic Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:12275-12284.
[13]GOODFELLOW I J,ABADIE J P,MIRZA M,et al.Generative adversarial nets[C]//Advances in Neural Information Proces-sing Systems.2014:2672-2680.
[14]MA J Y,YU W,LIANG P W,et al.FusionGAN:A generative adversarial network for infrared and visible image fusion[J].Information Fusion,2019,48:11-26.
[15]HUANG G,LIU Z,VAN D M L,et al.Densely connected con-volutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4700-4708.
[16]LI H,WU X J.Densefuse:A fusion approach to infrared and vi-sible images[J].IEEE Transactions on Image Processing,2018,28(5):2614-2623.
[17]LYU Q,SHAN H,STEBER C,et al.Multi-contrast super-resolution mri through a progressive network[J].IEEE Transactions on Medical Imaging,2020,39(9):2738-2749.
[18]GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improvedtraining of wasserstein gans[C]//Advances in Neural Information Processing Systems.2017:5767-5777.
[19]HAN Z.Research on multi-band image adaptive fusion method based on deep learning [D].Taiyuan:North University of China,2018.
[20]MOONON A U,HU J.Multi-focus image fusion based on NSCT and NSST[J].Sensing and Imaging,2015,16(1):4.
[1] 饶志双, 贾真, 张凡, 李天瑞.
基于Key-Value关联记忆网络的知识图谱问答方法
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[2] 汤凌韬, 王迪, 张鲁飞, 刘盛云.
基于安全多方计算和差分隐私的联邦学习方案
Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy
计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[3] 张佳, 董守斌.
基于评论方面级用户偏好迁移的跨领域推荐算法
Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer
计算机科学, 2022, 49(9): 41-47. https://doi.org/10.11896/jsjkx.220200131
[4] 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺.
时序知识图谱表示学习
Temporal Knowledge Graph Representation Learning
计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[5] 王剑, 彭雨琦, 赵宇斐, 杨健.
基于深度学习的社交网络舆情信息抽取方法综述
Survey of Social Network Public Opinion Information Extraction Based on Deep Learning
计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[6] 郝志荣, 陈龙, 黄嘉成.
面向文本分类的类别区分式通用对抗攻击方法
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[7] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
基于改进位置编码的谣言检测模型
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[8] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[9] 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木.
中文预训练模型研究进展
Advances in Chinese Pre-training Models
计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[10] 周慧, 施皓晨, 屠要峰, 黄圣君.
基于主动采样的深度鲁棒神经网络学习
Robust Deep Neural Network Learning Based on Active Sampling
计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[11] 杜航原, 李铎, 王文剑.
一种面向电商网络的异常用户检测方法
Method for Abnormal Users Detection Oriented to E-commerce Network
计算机科学, 2022, 49(7): 170-178. https://doi.org/10.11896/jsjkx.210600092
[12] 苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫.
小样本雷达辐射源识别的深度学习方法综述
Survey of Deep Learning for Radar Emitter Identification Based on Small Sample
计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138
[13] 胡艳羽, 赵龙, 董祥军.
一种用于癌症分类的两阶段深度特征选择提取算法
Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification
计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[14] 戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮.
基于DNGAN的磁共振图像超分辨率重建算法
Super-resolution Reconstruction of MRI Based on DNGAN
计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105
[15] 程成, 降爱莲.
基于多路径特征提取的实时语义分割方法
Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction
计算机科学, 2022, 49(7): 120-126. https://doi.org/10.11896/jsjkx.210500157
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!