Computer Science ›› 2019, Vol. 46 ›› Issue (9): 259-264.doi: 10.11896/j.issn.1002-137X.2019.09.039

• Graphics,Image & Pattern Recognition • Previous Articles     Next Articles

Image Localized Style Transfer Based on Convolutional Neural Network

MIAO Yong-wei1,2, LI Gao-yi1, BAO Chen1, ZHANG Xu-dong1, PENG Si-long3   

  1. (College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China)1;
    (College of Information Science and Technology,Zhejiang Sci-Tech University,Hangzhou 310018,China)2;
    (Institute of Automation,Chinese Academy of Sciences,Beijing 100190,China)3
  • Received:2018-07-20 Online:2019-09-15 Published:2019-09-02

Abstract: Image style transfer is a research hot topic in computer graphics and computer vision.Aiming at the difficulty in the style transfer of the local area of the content image in the existing image style transfer method,this paper proposed a localized image transfer framework based on convolutional neural network.First,according to the input content image and style image,the image style transfer network is used to generate the whole style transferred image.Then,the image foreground and the background area are determined by the mask generated by automatic semantic segmentation.Finally,according to style transfer result of the foreground or the background region,an image fusion algorithm based on Manhattan distance is proposed to optimize the convergence and smooth transition between the stylized object and the original area.The framework comprehensively considers the pixel values and positions of the target area and the boundary band,and experiments on three public image datasets demonstrate that the method can efficiently,quickly and naturally implement local style transfer of input content maps,and produce visual effects that are both artistic and authentic.

Key words: Localized image style transfer, Deep learning, Convolutional neural network (CNN), Manhattan distance, Automatic semantic segmentation

CLC Number: 

  • TP391
[1]GOOCH B,GOOCH A.Non-photorealistic rendering [M].New York:AK Peters/CRC Press,2001.
[2]DECARLO D,SANTELL A.Stylization and abstraction of photographs [J].ACM Transactions on Graphics,2002,21(3):769-776.
[3]WANG J,XU Y,SHUM H Y,et al.Video toning [J].ACM Transactions on Graphics,2004,23(3):574-583.
[4]LI P,SUN H Q,SHENG B,et al.Image stylization with enhanced structure on GPU [J].Science China Information Scie-nces,2012,55(5):1093-1105.
[5]ZANG Y,HUANG H,LI C F.Stroke style analysis for painterly rendering [J].Journal of Computer Science and Technology,2013,28(5):762-775.
[6]GATYS L A,Ecker A S,BETHGE M.Image style transfer using convolutional neural networks [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:2414-2423.
[7]GATYS L A,BETHGE M,HERTZMANN A,et al.Preserving color in neural artistic style transfer[J].arXiv:1606.05897,2016.
[8]JOHNSON J,ALAHI A,LI F F.Perceptual losses for real-time style transfer and super-resolution [C]//Proceedings of Euro-pean Conference on Computer Vision.Cham:Springer,2016:694-711.
[9]DUMOULIN V,SHLENS J,KUDLUR M,et al.A learned representation for artistic style [J].arXiv:1610.07629,2016.
[10]SHELHAMER E,LONG J,DARRELL T.Fully convolutional networks for semantic segmentation [J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,39(4):640-651.
[11]LAFFERTY J D,MCCALLUM A,PEREIRA F C N.Condi-tional random fields:probabilistic models for segmenting and labeling sequence data [C]//Proceedings of Eighteenth International Confe-rence on Machine Learning.Morgan Kaufmann Publishers Inc.,2001:282-289.
[12]DENG J,DONG W,SOCHER R,et al.ImageNet:A large-scale hierarchical image database [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2009:710-719.
[13]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition [J].arXiv:1409.1556,2014.
[14]HE K,ZHANG X,REN S,et al.Deep residual learning for ima-ge recognition [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:770-778.
[15]MALKAUTHEKAR M D.Analysis of euclidean distance andManhattan Distance measure in face recognition [C]//International Conference on Computational Intelligence and Information Technology.IET,2013:503-507.
[16]ABADI M,AGARWAL A,BARHAM P,et al.Tensorflow:large-scale machine learning on heterogeneous distributed systems [J].arXiv:1603.04467,2016.
[1] WANG Rui-ping, JIA Zhen, LIU Chang, CHEN Ze-wei, LI Tian-rui. Deep Interest Factorization Machine Network Based on DeepFM [J]. Computer Science, 2021, 48(1): 226-232.
[2] YU Wen-jia, DING Shi-fei. Conditional Generative Adversarial Network Based on Self-attention Mechanism [J]. Computer Science, 2021, 48(1): 241-246.
[3] TONG Xin, WANG Bin-jun, WANG Run-zheng, PAN Xiao-qin. Survey on Adversarial Sample of Deep Learning Towards Natural Language Processing [J]. Computer Science, 2021, 48(1): 258-267.
[4] DING Yu, WEI Hao, PAN Zhi-song, LIU Xin. Survey of Network Representation Learning [J]. Computer Science, 2020, 47(9): 52-59.
[5] HE Xin, XU Juan, JIN Ying-ying. Action-related Network:Towards Modeling Complete Changeable Action [J]. Computer Science, 2020, 47(9): 123-128.
[6] YE Ya-nan, CHI Jing, YU Zhi-ping, ZHAN Yu-liand ZHANG Cai-ming. Expression Animation Synthesis Based on Improved CycleGan Model and Region Segmentation [J]. Computer Science, 2020, 47(9): 142-149.
[7] DENG Liang, XU Geng-lin, LI Meng-jie, CHEN Zhang-jin. Fast Face Recognition Based on Deep Learning and Multiple Hash Similarity Weighting [J]. Computer Science, 2020, 47(9): 163-168.
[8] BAO Yu-xuan, LU Tian-liang, DU Yan-hui. Overview of Deepfake Video Detection Technology [J]. Computer Science, 2020, 47(9): 283-292.
[9] YUAN Ye, HE Xiao-ge, ZHU Ding-kun, WANG Fu-lee, XIE Hao-ran, WANG Jun, WEI Ming-qiang, GUO Yan-wen. Survey of Visual Image Saliency Detection [J]. Computer Science, 2020, 47(7): 84-91.
[10] WANG Wen-dao, WANG Run-ze, WEI Xin-lei, QI Yun-liang, MA Yi-de. Automatic Recognition of ECG Based on Stacked Bidirectional LSTM [J]. Computer Science, 2020, 47(7): 118-124.
[11] LIU Yan, WEN Jing. Complex Scene Text Detection Based on Attention Mechanism [J]. Computer Science, 2020, 47(7): 135-140.
[12] ZHANG Zhi-yang, ZHANG Feng-li, TAN Qi, WANG Rui-jin. Review of Information Cascade Prediction Methods Based on Deep Learning [J]. Computer Science, 2020, 47(7): 141-153.
[13] JIANG Wen-bin, FU Zhi, PENG Jing, ZHU Jian. 4Bit-based Gradient Compression Method for Distributed Deep Learning System [J]. Computer Science, 2020, 47(7): 220-226.
[14] CHEN Jin-yin, ZHANG Dun-Jie, LIN Xiang, XU Xiao-dong and ZHU Zi-ling. False Message Propagation Suppression Based on Influence Maximization [J]. Computer Science, 2020, 47(6A): 17-23.
[15] CHENG Zhe, BAI Qian, ZHANG Hao, WANG Shi-pu and LIANG Yu. Improving Hi-C Data Resolution with Deep Convolutional Neural Networks [J]. Computer Science, 2020, 47(6A): 70-74.
Full text



[1] LEI Li-hui and WANG Jing. Parallelization of LTL Model Checking Based on Possibility Measure[J]. Computer Science, 2018, 45(4): 71 -75 .
[2] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[3] ZHANG Jia-nan and XIAO Ming-yu. Approximation Algorithm for Weighted Mixed Domination Problem[J]. Computer Science, 2018, 45(4): 83 -88 .
[4] WU Jian-hui, HUANG Zhong-xiang, LI Wu, WU Jian-hui, PENG Xin and ZHANG Sheng. Robustness Optimization of Sequence Decision in Urban Road Construction[J]. Computer Science, 2018, 45(4): 89 -93 .
[5] SHI Wen-jun, WU Ji-gang and LUO Yu-chun. Fast and Efficient Scheduling Algorithms for Mobile Cloud Offloading[J]. Computer Science, 2018, 45(4): 94 -99 .
[6] ZHOU Yan-ping and YE Qiao-lin. L1-norm Distance Based Least Squares Twin Support Vector Machine[J]. Computer Science, 2018, 45(4): 100 -105 .
[7] LIU Bo-yi, TANG Xiang-yan and CHENG Jie-ren. Recognition Method for Corn Borer Based on Templates Matching in Muliple Growth Periods[J]. Computer Science, 2018, 45(4): 106 -111 .
[8] GENG Hai-jun, SHI Xin-gang, WANG Zhi-liang, YIN Xia and YIN Shao-ping. Energy-efficient Intra-domain Routing Algorithm Based on Directed Acyclic Graph[J]. Computer Science, 2018, 45(4): 112 -116 .
[9] CUI Qiong, LI Jian-hua, WANG Hong and NAN Ming-li. Resilience Analysis Model of Networked Command Information System Based on Node Repairability[J]. Computer Science, 2018, 45(4): 117 -121 .
[10] WANG Zhen-chao, HOU Huan-huan and LIAN Rui. Path Optimization Scheme for Restraining Degree of Disorder in CMT[J]. Computer Science, 2018, 45(4): 122 -125 .