Computer Science

Computer Science ›› 2023, Vol. 50 ›› Issue (8): 133-141.doi: 10.11896/jsjkx.220600065

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Remote Sensing Image Pan-sharpening Method Based on Generative Adversarial Network

YAN Yan1, SUI Yi1, SI Jianwei2   

  1. 1 College of Computer Science and Technology,Qingdao University,Qingdao,Shandong 266071,China
    2 Faculty of Information Science and Engineering,Ocean University of China,Qingdao,Shandong 266100,China
  • Received:2022-06-07 Revised:2022-10-11 Online:2023-08-15 Published:2023-08-02
  • About author:YAN Yan,born in 1996,master candidate.Her main research interests include deep learning and image proces-sing.
    SUI Yi,born in 1984,Ph.D,associate professor,master supervisor.Her main research interests include big data mo-deling and analysis,artifical intelligence and image processing.
  • Supported by:
    Young Scientists Fund of the National Natural Science Foundation of China(41706198).

PDF (PC)

264

Abstract: Remote sensing image pan-sharpening methods are generally based on Wald protocol,resulting in blurred texture details,colors and ambiguous boundaries of the reconstructed images.To solve the problem,a remote sensing image pan-sharpening method based on generative adversarial networks(GAN),PAN-GAN,is proposed in this paper.The multispectral image is employed as the reference image.The grayscale reference image is applied to simulate the panchromatic image and the blurred reference image is adpoted as input of the generator.The generator extracts the texture details of the grayscale reference image and spectral features of the blurred reference image for the fusion reconstruction.Meanwhile,the perceptual loss is introduced to optimize the reconstruction results with adversarial loss and pixel loss,so that the reconstructed images have spectral and texture detail features closer to the reference image.Experiments are carried out on the datasets of three remote sensing satellites including QuickBird,GaoFen-2 and WorldView-2.The results show that the reconstructed images obtained by PAN-GAN have more realistic spectral and spatial texture details compared with common methods.The usage of grayscale reference images can significantly improve the performance of the original method,and the average grayscale improvement is the most obvious.The perceptual loss can further optimize the reconstruction results and verify the effectiveness of the proposed method.

Key words: Remote sensing images, Pan-sharpening, Generative adversarial networks, Perceptual features

CLC Number: 

  • TP391
[1]WANG X,YONG L,JIANG Z.An IHS Fusion Method based on Structural Similarity[J].Remote Sensing Technology and Application,2011,26(5):670-676.
[2]PING S,LEI D,JUAN N.Multi-scale Remote Sensing ImageFusion Method based on Region Segmentation[J].Remote Sen-sing Technology and Application,2013,27(6):844-849.
[3]ZENG D,HU Y,HUANG Y,et al.Pan-sharpening with structural consistency and 1/2 gradient prior[J].Remote Sensing Letters,2016,7(12):1170-1179.
[4]GILLESPIE A R,KAHLE A B,WALKER R E.Color enhancement of highly correlated images.II.Channel ratio and “chromaticity” transformation techniques[J].Remote Sensing of Environment,1987,22(3):343-365.
[5]PADWICK C,DESKEVICH M,PACIFICI F,et al.WorldView-2 pan-sharpening[C]//Proceedings of the ASPRS 2010 Annual Conference.San Diego,CA,USA,2010:1-14.
[6]FENG J,ZEREN L,XIA C,et al.Remote sensing image fusion method based on PCA and NSCT transform[J].Journal of Graphics,2017,38(2):247-252.
[7]RAHMANI S,STRAIT M,MERKURJEV D,et al.An adaptive IHS pan-sharpening method[J].IEEE Geoscience and Remote Sensing Letters,2010,7(4):746-750.
[8]LIU J,LI X,ZHU K,et al.Distributed Compressed SensingBased Remote Sensing Image Fusion Algorithm[J].Journal of Electronics & Information Technology,2017,39(10):2374-2381.
[9]RANCHIN T,WALD L.Fusion of high spatial and spectral re-solution images:The ARSIS concept and its implementation[J].Photogrammetric Engineering and Remote Sensing,2000,66(1):49-61.
[10]GIUSEPPE M,DAVIDE C,LUISA V,et al.Pansharpening by Convolutional Neural Networks[J].Remote Sensing,2016,8(7):594.
[11]SCARPA G,VITALE S,COZZOLINO D.Target-adaptiveCNN-based pansharpening[J].IEEE Transactions on Geo-science and Remote Sensing,2018,56(9):5443-5457.
[12]YANG J,FU X,HU Y,et al.PanNet:A Deep Network Architecture for Pan-Sharpening[C]//2017 IEEE International Conference on Computer Vision(ICCV).IEEE,2017:5448-5457.
[13]DONG W Q,ZHANG T Z,QU J H,et al.Laplacian Pyramid Dense Network for Hyperspectral Pansharpening[J].IEEE Transactions on Geoscience and Remote Sensing,2022,(60):1-13.
[14]LEI D J,DU J H,ZHANG L P,et al.Multi-stream Architecture and Multi-scale Convolutional Neural Network for Remote Sensing Image Fusion[J].Journal of Electronics & Information Technology,2022,44(1):8.
[15]AZARANG A,MANOOCHEHRI H E,KEHTARNAVAZ N.Convolutional autoencoder-based multispectral image fusion[J].IEEE Access,2019,7:35673-35683.
[16]WANG L,YANG X M.Remote Sensing Image Pansharpening Feedback Network Based on Perceptual Loss[J].Computer Science,2021,48(8):91-98.
[17]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Ge-nerative Adversarial Nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2.2014:2672-2680.
[18]DONG W Q,HOU S X,XIAO S,et al.Generative Dual-Adversarial Network With Spectral Fidelity and Spatial Enhancement for Hyperspectral Pansharpening[J].IEEE Transactions on Neural Networks and Learning Systems,2021 (99):1-15.
[19]YANG J,LI W J,WANG W G,et al.Generative adversarial network for image super-resolution combining perceptual loss[J].Journal of Image and Graphics,2019,24(8):1270-1282.
[20]MA J,XU H,JIANG J,et al.DDcGAN:A dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J].IEEE Transactions on Image Processing,2020,29:4980-4995.
[21]LIU Q,ZHOU H,XU Q,et al.PSGAN:A generative adversa-rial network for remote sensing image pan-sharpening[J].IEEE Transactions on Geoscience and Remote Sensing,2020,59(12):10227-10242.
[22]OZCELIK F,ALGANCI U,SERTEL E,et al.Rethinking CNN-based pansharpening:Guided colorization of panchromatic images via GANs[J].IEEE Transactions on Geoscience and Remote Sensing,2020,59(4):3486-3501.
[23]WALD L.Quality of high resolution synthesised images:Isthere a simple criterion?[C]//Proceedings of the Third Conference “Fusion of Earth Data:Merging Point Measurements,Raster Maps and Remotely Sensed Images”.2000:99-103.
[24]JOLICOEUR-MARTINEAU A.The relativistic discriminator:a key element missing from standard GAN[J].arXiv:1807,00734,2018.
[25]LIN M,CHEN Q,YAN S.Network in network[J].arXiv:1312,4400,2013.
[26]NAIR V,HINTON G E.Rectified Linear Units Improve Re-stricted Boltzmann Machines Vinod Nair[C]//Proceedings of the 27th International Conference on Machine Learning (ICML-10).Haifa,Israel,2010:807-814.
[27]XU B,WANG N,CHEN T,et al.Empirical evaluation of rectified activations in convolutional network[J].arXiv:1505.00853,2015.
[28]ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,HI,USA:IEEE Press,2017:1125-1134.
[29]YUHAS R H,GOETZ A F,BOARDMAN J W.Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm[C]//Proceedings of the JPL,Summaries of the Third Annual JPL Airborne Geoscience Workshop Volume 1:AVIRIS Workshop.1992:147-149.
[30]ZHOU J,CIVCO D L,SILANDER J.A wavelet transformmethod to merge Landsat TM and SPOT panchromatic data[J].International Journal of Remote Sensing,1998,19(4):743-757.
[31]WANG Z,BOVIK A C.A universal image quality index[J].IEEE Signal Processing Letters,2002,9(3):81-84.
[32]ZHANG R,ISOLA P,EFROS A A,et al.The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:586-595.
[33]ALPARONE L,AIAZZI B,BARONTI S,et al.Multispectraland Panchromatic Data Fusion Assessment Without Reference[J].Photogrammetric Engineering & Remote Sensing,2008,74(2):193-200.
[1] WANG Jinwei, ZENG Kehui, ZHANG Jiawei, LUO Xiangyang, MA Bin. GAN-generated Face Detection Based on Space-Frequency Convolutional Neural Network [J]. Computer Science, 2023, 50(6): 216-224.
[2] WANG Kun-shu, ZHANG Ze-hui, GAO Tie-gang. Reversible Hidden Algorithm for Remote Sensing Images Based on Hachimoji DNA and QR Decomposition [J]. Computer Science, 2022, 49(8): 127-135.
[3] XU Guo-ning, CHEN Yi-peng, CHEN Yi-ming, CHEN Jin-yin, WEN Hao. Data Debiasing Method Based on Constrained Optimized Generative Adversarial Networks [J]. Computer Science, 2022, 49(6A): 184-190.
[4] XU Hui, KANG Jin-meng, ZHANG Jia-wan. Digital Mural Inpainting Method Based on Feature Perception [J]. Computer Science, 2022, 49(6): 217-223.
[5] DOU Zhi, WANG Ning, WANG Shi-jie, WANG Zhi-hui, LI Hao-jie. Sketch Colorization Method with Drawing Prior [J]. Computer Science, 2022, 49(4): 195-202.
[6] GAO Zhi-yu, WANG Tian-jing, WANG Yue, SHEN Hang, BAI Guang-wei. Traffic Prediction Method for 5G Network Based on Generative Adversarial Network [J]. Computer Science, 2022, 49(4): 321-328.
[7] LI Si-quan, WAN Yong-jing, JIANG Cui-ling. Multiple Fundamental Frequency Estimation Algorithm Based on Generative Adversarial Networks for Image Removal [J]. Computer Science, 2022, 49(3): 179-184.
[8] TAN Xin-yue, HE Xiao-hai, WANG Zheng-yong, LUO Xiao-dong, QING Lin-bo. Text-to-Image Generation Technology Based on Transformer Cross Attention [J]. Computer Science, 2022, 49(2): 107-115.
[9] ZHOU Shi-jin, XING Hong-jieHebei. Memory-augmented GAN-based Anomaly Detection [J]. Computer Science, 2022, 49(11A): 211000202-9.
[10] ZHANG Wei-qi, TANG Yi-feng, LI Lin-yan, HU Fu-yuan. Image Stream From Paragraph Method Based on Scene Graph [J]. Computer Science, 2022, 49(1): 233-240.
[11] XU Tao, TIAN Chong-yang, LIU Cai-hua. Deep Learning for Abnormal Crowd Behavior Detection:A Review [J]. Computer Science, 2021, 48(9): 125-134.
[12] LIN Zhen-xian, ZHANG Meng-kai, WU Cheng-mao, ZHENG Xing-ning. Face Image Inpainting with Generative Adversarial Network [J]. Computer Science, 2021, 48(9): 174-180.
[13] PAN Xiao-qin, LU Tian-liang, DU Yan-hui, TONG Xin. Overview of Speech Synthesis and Voice Conversion Technology Based on Deep Learning [J]. Computer Science, 2021, 48(8): 200-208.
[14] ZHANG Man, LI Jie, ZHU Xin-zhong, SHEN Ji, CHENG Hao-tian. Augmentation Technology of Remote Sensing Dataset Based on Improved DCGAN Algorithm [J]. Computer Science, 2021, 48(6A): 80-84.
[15] WANG Jian-ming, LI Xiang-feng, YE Lei, ZUO Dun-wen, ZHANG Li-ping. Medical Image Deblur Using Generative Adversarial Networks with Channel Attention [J]. Computer Science, 2021, 48(6A): 101-106.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.