Computer Science ›› 2021, Vol. 48 ›› Issue (8): 91-98.doi: 10.11896/jsjkx.200700112

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Remote Sensing Image Pansharpening Feedback Network Based on Perceptual Loss

WANG Le, YANG Xiao-min   

  1. College of Electronics and Information Engineering,Sichuan University,Chengdu 610065,China
  • Received:2020-07-19 Revised:2020-08-18 Published:2021-08-10
  • About author:WANG Le,born in 1995,postgraduate.Her main research interests include remote sensing image processing.(sxwangll@163.com)YANG Xiao-min,born in 1980,professor,Ph.D supervisor.Her main research interests include image process and pattern recognition.
  • Supported by:
    National Natural Science Foundation of China(61701327).

Abstract: Pansharpening aims to sharpen a low-resolution multi-channel multispectral(MS) image from a high-resolution single-channel panchromatic(PAN) image to obtain a high resolution multispectral(HRMS) image,which is an important task in remote sensing image processing.A feedback network based on perceptual loss is proposed.First,the detail information and the spectral information are extracted from the PAN image and the MS image respectively,and then they are combined to use the stacked up-and down-sampling layers and dense connections for information fusion.The feedback connection is used to enrich the low-level information with high-level information.Finally,the HRMS image is reconstructed.Compared with the traditional pansharpening algorithms,the proposed algorithm uses the PAN image and the HRMS image as the supervision of the network,and the output image contains richer spatial detail information by obtaining the perceptual loss of the PAN image and the network reconstructed HRMS image.The experimental results show that the proposed algorithm has better results than the widely used algorithms both in objective evaluation and visual perception.

Key words: Convolutional neural network, Feedback, Pansharpening, Perceptual loss

CLC Number: 

  • TP751.1
[1]CHEN J Y,TIAN J Q.Vegetation Classification Based on High-resolution Satellite Image [J].Journal of Remote Sensing,2007,11(2):221-227.
[2]WANG X Y,LIU Y,JIANG Z Y.An IHS Fusion Method based on Structural Similarity[J].Remote Sensing Technology and Application,2011,26(5):670-676.
[3]ZENG Y Y,HE J N.Remote Sensing Image Fusion MethodBased on Regional Wavelet Statistical Features[J].Computer Engineering,2011,37(19):198-200.
[4]SUN P,DENG L,NIE J.Multi-scale Remote Sensing Image Fusion Method based on Region Segmentation[J].Remote Sensing Technology and Application,2012,27(6):844-849.
[5]LI S T,YANG B.A new pan-sharpening method using a compressed sensing technique[J].IEEE Transactions on Geoscience and Remote Sensing,2011,49(2):738-746.
[6]YIN H T.Sparse representation based pan-sharpening with details injection model[J].Signal Processing,2015,113:218-227.
[7]MASI G,COZZOLINO D,VERDOLIVA L,,et al.Pansharpening by Convolutional Neural Networks[J].Remote Sensing,2016,8(7):594-616.
[8]ZHANG Y,LIU C,SUN M,et al.Pan-Sharpening Using an Efficient Bidirectional Pyramid Network[J].IEEE Transactions on Geoscience and Remote Sensing,2019,57(8):5549-5563.
[9]LIU X,LIU Q,WANG Y.Remote Sensing Image Fusion Based on Two-stream Fusion Network[J].Lecture Notes in Computer Science,2018,10704:428-439.
[10]GUO Y,YE F,GONG H.Learning an Efficient ConvolutionNeural Network for Pan-sharpening[J].Algorithms,2019,12(1):16.
[11]YAO W,ZENG Z G,LIAN C,et al.Pixel-wise regression using U-Net and its application on pan-sharpening[J].Neurocompu-ting,2018,312:364-371.
[12]HE L,RAO Y,LI J,et al.Pansharpening via Detail Injection Based Convolutional Neural Net-works[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sen-sing,2019,12(4):1188-1204.
[13]CARREIRA J,AGRAWAL P,FRAGKIADAKI K,et al.Human Pose Estimation with Iterative Error Feedback[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE Press,2016:4733-4742.
[14]HARIS M,SHAKHNAROVICH G,UKITA N.Deep back-projection networks for super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Utah:IEEE Press,2018:1664-1673.
[15]HAN W,CHANG S,LIU D,et al.Image Super-Resolution via Dual-State Recurrent Networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Utah:IEEE Press,2018:1654-1663.
[16]LI Z,YANG J,LIU Z,et al.Feedback Network for Image Super-Resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.California:IEEE Press,2019:3867-3876.
[17]JOHNSON J,ALAHI A,LI F F.Perceptual Losses for Real-Time Style Transfer and Super-Resolution[J].Lecture Notes in Computer Science,2016,9906:694-711.
[18]WALD L,RANCHIN T,MANGOLINI M.Fusion of satelliteimages of different spatial resolutions:Assessing the quality of resulting images[J].Photogrammetric Engineering and Remote Sensing,1997,63(6):691-699.
[19]GATYS L A,ECKER A S,BETHGE M.Image Style Transfer Using Convolutional Neural Networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE Press,2016:2414-2423.
[20]KANG X,LI S,BENEDIKTSSON J A.Pansharpening withMatting Model[J].IEEE Transactions on Geoence and Remote Sensing,2014,52(8):5088-5099.
[21]JIN B,KIM G,CHO N I.Wavelet-domain satellite image fusion based on a generalized fusion equation[J].Journal of Applied Remote Sensing,2014,8(1):080599.
[22]TU T M,HUANG P S,HUNG C L,et al.A Fast Intensity-Hue-Saturation Fusion Technique With Spectral Adjustment for IKONOS Imagery[J].IEEE Geoence and Remote Sensing Letters,2004,1(4):309-312.
[23]JR P S C,KWARTENG A Y.Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis[J].Photogrammetric Engineering and Remote Sensing,1989,55(3):339-348.
[24]SHAH V P,YOUNAN N H,KING R L.An Efficient Pan-Sharpening Method via a Combined Adaptive PCA Approach and Contourlets[J].IEEE Transactions on Geoence & Remote Sensing,2008,46(5):1323-1335.
[25]WEI Y,YUAN Q,SHEN H,et al.Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network[J].IEEE Geoence and Remote Sensing Letters,2017,14(10):1795-1799.
[1] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[2] CHEN Yong-quan, JIANG Ying. Analysis Method of APP User Behavior Based on Convolutional Neural Network [J]. Computer Science, 2022, 49(8): 78-85.
[3] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[4] CHEN Jun, HE Qing, LI Shou-yu. Archimedes Optimization Algorithm Based on Adaptive Feedback Adjustment Factor [J]. Computer Science, 2022, 49(8): 237-246.
[5] DAI Zhao-xia, LI Jin-xin, ZHANG Xiang-dong, XU Xu, MEI Lin, ZHANG Liang. Super-resolution Reconstruction of MRI Based on DNGAN [J]. Computer Science, 2022, 49(7): 113-119.
[6] LIU Yue-hong, NIU Shao-hua, SHEN Xian-hao. Virtual Reality Video Intraframe Prediction Coding Based on Convolutional Neural Network [J]. Computer Science, 2022, 49(7): 127-131.
[7] XU Ming-ke, ZHANG Fan. Head Fusion:A Method to Improve Accuracy and Robustness of Speech Emotion Recognition [J]. Computer Science, 2022, 49(7): 132-141.
[8] JIN Fang-yan, WANG Xiu-li. Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM [J]. Computer Science, 2022, 49(7): 179-186.
[9] ZHANG Jia-hao, LIU Feng, QI Jia-yin. Lightweight Micro-expression Recognition Architecture Based on Bottleneck Transformer [J]. Computer Science, 2022, 49(6A): 370-377.
[10] WANG Jian-ming, CHEN Xiang-yu, YANG Zi-zhong, SHI Chen-yang, ZHANG Yu-hang, QIAN Zheng-kun. Influence of Different Data Augmentation Methods on Model Recognition Accuracy [J]. Computer Science, 2022, 49(6A): 418-423.
[11] SUN Jie-qi, LI Ya-feng, ZHANG Wen-bo, LIU Peng-hui. Dual-field Feature Fusion Deep Convolutional Neural Network Based on Discrete Wavelet Transformation [J]. Computer Science, 2022, 49(6A): 434-440.
[12] YANG Yue, FENG Tao, LIANG Hong, YANG Yang. Image Arbitrary Style Transfer via Criss-cross Attention [J]. Computer Science, 2022, 49(6A): 345-352.
[13] YANG Jian-nan, ZHANG Fan. Classification Method for Small Crops Combining Dual Attention Mechanisms and Hierarchical Network Structure [J]. Computer Science, 2022, 49(6A): 353-357.
[14] WU Zi-bin, YAN Qiao. Projected Gradient Descent Algorithm with Momentum [J]. Computer Science, 2022, 49(6A): 178-183.
[15] QING Chao-jin, DU Yan-hong, YE Qing, YANG Na, ZHANG Min-tao. Enhanced ELM-based Superimposed CSI Feedback Method with CSI Estimation Errors [J]. Computer Science, 2022, 49(6A): 632-638.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!