Computer Science ›› 2020, Vol. 47 ›› Issue (2): 126-134.doi: 10.11896/jsjkx.190100119

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Image Semantic Segmentation Based on Deep Feature Fusion

ZHOU Peng-cheng1,GONG Sheng-rong1,2,ZHONG Shan1,2,BAO Zong-ming1,DAI Xing-hua1   

  1. (School of Computer Science and Technology,Soochow University,Suzhou,Jiangsu 215006,China)1;
    (School of Computer Science and Engineering,Changshu Institute of Technology,Suzhou,Jiangsu 215500,China)2
  • Received:2019-01-15 Online:2020-02-15 Published:2020-03-18
  • About author:ZHOU Peng-cheng,born in 1992,postgraduate.His main research interests include digital image processing,computer vision and pattern recognition;GONG Sheng-rong,born in 1966,Ph.D,professor,Ph.D supervisor,is the vice chairman of Suzhou CCF Association.His main research interests include machine learning and computer vision.
  • Supported by:
    This work was supported by the National Natural Science Foundation of China (61272005, 61702055), Natural Science Foundation of Jiangsu Province (BK20151254, BK20151260), Six Peak Talents Projecof Jiangsu Province (DZXX-027) and Cloud Integration Science and Education Innovation Foundation of Ministry of Education Science and Technology Development Center (2017B03112).

Abstract: When feature extraction is performed by using convolutional networks in image semantic segmentation,the context information is lost due to the reduced resolution of features by the repeated combination of maximum pooling and downsampling operations,so that the segmentation result loses the sensitivity to the object location.Although the network based on the encoder-decoder architecture gradually refines the output precision through the jump connection in the process of restoring the resolution,the operation of simply summing the adjacent features ignores the difference between the features and easily leads to local mis-identification of objects and other issues.To this end,an image semantic segmentation method based on deep feature fusion was proposed.It adopts a network structure in which multiple sets of fully convolutional VGG16 models are combined in parallel,processes multi-scale images in the pyramid in parallel efficiently with atrous convolutions,extracts multi-level context feature,and fuses layer by layer through a top-down method to capture the context information as far as possible.At the same time,the layer-by-layer label supervision strategy based on the improved loss function is an auxiliary support with a dense conditional random field of pixels modeling in the backend,which has certain optimization in terms of the difficulty of model training and the accuracy of predictive output.Experimental data show that the image semantic segmentation algorithm improves the classification of target objects and the location of spatial details by layer-by-layer fusion of deep features that characterize different scale context information.The experimental results obtained on PASCAL VOC 2012 and PASCAL CONTEXT datasets show that the proposed method achieves mIoU accuracy of 80.5% and 45.93%,respectively.The experimental data fully demonstrate that deep feature extraction,feature layer-by-layer fusion and layer-by-layer label supervision strategy in the parallel framework can jointly optimize the algorithm architecture.The feature comparison shows that the model can capture rich context information and obtain more detailed image semantic features.Compared with similar methods,it has obvious advantages.

Key words: Image semantic segmentation, Deep feature, Atrous convolution, Feature fusion, Context information, Conditional random field

CLC Number: 

  • TP391
[1]LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Boston:IEEE Press,2015:3431-3440.
[2]YU F,KOLTUM V.Multi-Scale Context Aggregation by Dila-tedConvolutions[C]∥Proceedings of International Conference on Learning Representations.Puerto Rico:IEEE Press,2016:397-410.
[3]WANG P,CHEN P,YUAN Y,et al.Understanding Convolution for Semantic Segmentation[C]∥Proceedings of IEEE Winter Conference on Applications of Computer Vision.Santa Rosa:IEEE Press,2017:1451-1460.
[4]LIU Z,LI X,LUO P,et al.Semantic Image Segmentation via Deep Parsing Network[C]∥Proceedings of IEEE International Conference on Computer Vision.Santiago Chile:IEEE Press,2015:1377-1385.
[5]NGUYEN K,FOOKES C,SRIDHARAN S.Deep Context Mo-deling for Semantic Segmentation[C]∥Proceedings of IEEE Winter Conference on Applications of Computer Vision.Santa Rosa,California,United States:IEEE Press,2017:56-63.
[6]GHIASI G,FOWLKES C C.Laplacian Pyramid Reconstruction and Refinement for Semantic Segmentation[C]∥Proceedings of European Conference on Computer Vision.Cham:Springer Press,2016:519-534.
[7]BERTASIUS G,TORRESANI L,YU S X,et al.Convolutional Random Walk Networks for Semantic Image Segmentation[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii:IEEE Press,2017:6137-6145.
[8]DAI J,HE K,SUN J.BoxSup:Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation[C]∥Proceedings of IEEE International Conference on Computer Vision.Santiago,Chile:IEEE Press,2015:1635-1643.
[9]WANG G,LUO P,LIN L,et al.Learning Object Interactions and Descriptions for Semantic Image Segmentation[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,USA:IEEE Press,2017:5235-5243.
[10]MAURO D D,FURNARI A,PATANE G,et al.Scene Adaptation for Semantic Segmentation using Adversarial Learning[C]∥Proceedings of IEEE International Conference on Advanced Video and Signal Based Surveillance.Auckland,New Zealand:IEEE Press,2018:1-6.
[11]ZHANG Y H,QIU Z F,YAO T,et al.Fully Convolutional Adaptation Networks for Semantic Segmentation[C]∥Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE Press,2018:6810-6818.
[12]TSAI Y H,HUNG W C,Schulter S,et al.Learning to Adapt Structured Output Space for Semantic Segmentation[C]∥Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE Press,2018:7472-7481.
[13]LENG J X,LIU Y,ZHANG T L,et al.Context-Aware U-Net for Biomedical Image Segmentation[C]∥Proceedings of IEEE International Conference on Bioinformatics and Biomedicine.Madrid,Spain:IEEE Press,2018:2535-2538.
[14]BADRINARAYANAN V,KENDALL A,CIPOLLA R.Seg-Net:A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation [J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,39(12):2481-2495.
[15]BULO S R,NEUHOLD G,KONTSCHIEDER P.Loss Max-Pooling for Semantic Image Segmentation[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,USA:IEEE Press,2017:7082-7091.
[16]LIN G,SHEN C,HENGEL A V,et al.Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV,United States:IEEE Press,2016:3194-3203.
[17]LIN G,SHEN C,HENGEL A V,et al.Exploring Context with Deep Structured Models for Semantic Segmentation [J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2018,40(6):1352-1366.
[18]CHEN L C,PAPANDREOU G,KOKKINOS I,et al.DeepLab:Semantic Image Segmentation with Deep Convolutional Nets,Atrous Convolution,and Fully Connected CRFs [J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2016,40(4):834-848.
[19]PHILIPPK,KOLTUN V.Parameter Learning and Convergent Inference for Dense Random Fields[C]∥Proceedings of International Conference on International Conference on Machine Learning.Atlanta,GA,USA:ACM Press,2013:513-521.
[20]ADAMS A,BAEK J,DAVIS M A.Fast High-Dimensional Filtering Using the Permutohedral Lattice[J].Computer Graphics Forum,2010,29(2):753-762.
[21]EVERINGHAM M,ESLAMI S M A,Van G L,et al.The PASCAL Visual Object Classes Challenge:A Retrospective [J].International Journal of Computer Vision,2015,111(1):98-136.
[22]HARIHARAN B,BOURDEV L,ARBELAEZ P,MALIK J,et al.Semantic Contours from Inverse Detectors[C]∥Proceedings of IEEE International Conference on Computer Vision.Barcelona:IEEE Press,2011:991-998.
[23]MOTTAGHI R,CHEN X,LIU X,et al.The Role of Context for Object Detection and Semantic Segmentation in the Wild[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington DC:ACM Press,2014:891-898.
[24]ABDULNABI A H,SHUAI B,WINKLER S,et al.Episodic CAMN:Contextual Attention-Based Memory Networks with Iterative Feedback for Scene Labeling[C]∥Proceedings ofIEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,USA:IEEE Press,2018:6278-6287.
[25]WU Z,SHEN C,ANTONV D H.Bridging Category-Level and Instance-Level Semantic Image Segmentation[J].International Journal of Computer Vision,2016,111(1):140-155.
[26]ZHENG S,JAYASUMANA S,VINEET V,et al.Conditional Random Fields as Recurrent Neural Networks[C]∥Proceedings of IEEE International Conference on Computer Vision.Santiago,Chile:IEEE Press,2015:1529-1537.
[1] ZHANG Fan, HE Wen-qi, JI Hong-bing, LI Dan-ping, WANG Lei. Multi-view Dictionary-pair Learning Based on Block-diagonal Representation [J]. Computer Science, 2021, 48(1): 233-240.
[2] HUANG Yong-tao, YAN Hua. Scene Graph Generation Model Combining Attention Mechanism and Feature Fusion [J]. Computer Science, 2020, 47(6): 133-137.
[3] ZHU Wei, WANG Tu-qiang, CHEN Yue-feng, HE De-feng. Object-level Edge Detection Algorithm Based on Multi-scale Residual Network [J]. Computer Science, 2020, 47(6): 144-150.
[4] PEI Jia-zhen, XU Zeng-chun, HU Ping. Person Re -identification Fusing Viewpoint Mechanism and Pose Estimation [J]. Computer Science, 2020, 47(6): 164-169.
[5] HU Yu-jia, GAN Wei, ZHU Min. Enhancer-Promoter Interaction Prediction Based on Multi-feature Fusion [J]. Computer Science, 2020, 47(5): 64-71.
[6] CHEN Yan-wen,LI Kun,HAN Yan,WANG Yan-ping. Musical Note Recognition of Musical Instruments Based on MFCC and Constant Q Transform [J]. Computer Science, 2020, 47(3): 149-155.
[7] FU Xue-yang,SUN Qi,HUANG Yue,DING Xing-hao. Single Image De-raining Method Based on Deep Adjacently Connected Networks [J]. Computer Science, 2020, 47(2): 106-111.
[8] GENG Lei-lei, CUI Chao-ran, SHI Cheng, SHEN Zhen, YIN Yi-long, FENG Shi-hong. Social Image Tag and Group Joint Recommendation Based on Deep Multi-task Learning [J]. Computer Science, 2020, 47(12): 177-182.
[9] DUAN Jian-yong, YOU Shi-xin, ZHANG Mei, WANG Hao. Keyword Extraction Based on Multi-feature Fusion [J]. Computer Science, 2020, 47(11A): 73-77.
[10] MA Kang, LOU Jing-tao, SU Zhi-yuan, LI Yong-le, ZHU Yuan. Object Tracking Algorithm Based on Feature Fusion and Adaptive Scale Kernel Correlation Filter [J]. Computer Science, 2020, 47(11A): 224-230.
[11] LI Hao-xiang, LI Hao-jun. MACTEN:Novel Large Scale Cloth Texture Classification Architecture [J]. Computer Science, 2020, 47(11A): 258-265.
[12] YANG Pei-jian, WU Xiao-fu, ZHANG Suo-fei, ZHOU Quan. Semantic Segmentation Transfer Algorithm Based on Atrous Convolution Discriminator [J]. Computer Science, 2020, 47(11): 174-178.
[13] FAN Wei, LIU Ting, HUANG Rui, GUO Qing, ZHANG Bao. Low-level CNN Feature Aided Image Instance Segmentation [J]. Computer Science, 2020, 47(11): 186-191.
[14] XU Yang,WANG Jian-cheng,LIU Qi-yuan,LI Shou-shan. Intention Detection in Spoken Language Based on Context Information [J]. Computer Science, 2020, 47(1): 205-211.
[15] JIN Kun, CHEN Shao-chang. Status and Development of Gait Recognition [J]. Computer Science, 2019, 46(6A): 30-34.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] LEI Li-hui and WANG Jing. Parallelization of LTL Model Checking Based on Possibility Measure[J]. Computer Science, 2018, 45(4): 71 -75 .
[2] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[3] ZHANG Jia-nan and XIAO Ming-yu. Approximation Algorithm for Weighted Mixed Domination Problem[J]. Computer Science, 2018, 45(4): 83 -88 .
[4] WU Jian-hui, HUANG Zhong-xiang, LI Wu, WU Jian-hui, PENG Xin and ZHANG Sheng. Robustness Optimization of Sequence Decision in Urban Road Construction[J]. Computer Science, 2018, 45(4): 89 -93 .
[5] SHI Wen-jun, WU Ji-gang and LUO Yu-chun. Fast and Efficient Scheduling Algorithms for Mobile Cloud Offloading[J]. Computer Science, 2018, 45(4): 94 -99 .
[6] ZHOU Yan-ping and YE Qiao-lin. L1-norm Distance Based Least Squares Twin Support Vector Machine[J]. Computer Science, 2018, 45(4): 100 -105 .
[7] LIU Bo-yi, TANG Xiang-yan and CHENG Jie-ren. Recognition Method for Corn Borer Based on Templates Matching in Muliple Growth Periods[J]. Computer Science, 2018, 45(4): 106 -111 .
[8] GENG Hai-jun, SHI Xin-gang, WANG Zhi-liang, YIN Xia and YIN Shao-ping. Energy-efficient Intra-domain Routing Algorithm Based on Directed Acyclic Graph[J]. Computer Science, 2018, 45(4): 112 -116 .
[9] CUI Qiong, LI Jian-hua, WANG Hong and NAN Ming-li. Resilience Analysis Model of Networked Command Information System Based on Node Repairability[J]. Computer Science, 2018, 45(4): 117 -121 .
[10] WANG Zhen-chao, HOU Huan-huan and LIAN Rui. Path Optimization Scheme for Restraining Degree of Disorder in CMT[J]. Computer Science, 2018, 45(4): 122 -125 .