Computer Science ›› 2019, Vol. 46 ›› Issue (9): 259-264.doi: 10.11896/j.issn.1002-137X.2019.09.039

• Graphics,Image & Pattern Recognition • Previous Articles     Next Articles

Image Localized Style Transfer Based on Convolutional Neural Network

MIAO Yong-wei1,2, LI Gao-yi1, BAO Chen1, ZHANG Xu-dong1, PENG Si-long3   

  1. (College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China)1;
    (College of Information Science and Technology,Zhejiang Sci-Tech University,Hangzhou 310018,China)2;
    (Institute of Automation,Chinese Academy of Sciences,Beijing 100190,China)3
  • Received:2018-07-20 Online:2019-09-15 Published:2019-09-02

Abstract: Image style transfer is a research hot topic in computer graphics and computer vision.Aiming at the difficulty in the style transfer of the local area of the content image in the existing image style transfer method,this paper proposed a localized image transfer framework based on convolutional neural network.First,according to the input content image and style image,the image style transfer network is used to generate the whole style transferred image.Then,the image foreground and the background area are determined by the mask generated by automatic semantic segmentation.Finally,according to style transfer result of the foreground or the background region,an image fusion algorithm based on Manhattan distance is proposed to optimize the convergence and smooth transition between the stylized object and the original area.The framework comprehensively considers the pixel values and positions of the target area and the boundary band,and experiments on three public image datasets demonstrate that the method can efficiently,quickly and naturally implement local style transfer of input content maps,and produce visual effects that are both artistic and authentic.

Key words: Localized image style transfer, Deep learning, Convolutional neural network (CNN), Manhattan distance, Automatic semantic segmentation

CLC Number: 

  • TP391
[1]GOOCH B,GOOCH A.Non-photorealistic rendering [M].New York:AK Peters/CRC Press,2001.
[2]DECARLO D,SANTELL A.Stylization and abstraction of photographs [J].ACM Transactions on Graphics,2002,21(3):769-776.
[3]WANG J,XU Y,SHUM H Y,et al.Video toning [J].ACM Transactions on Graphics,2004,23(3):574-583.
[4]LI P,SUN H Q,SHENG B,et al.Image stylization with enhanced structure on GPU [J].Science China Information Scie-nces,2012,55(5):1093-1105.
[5]ZANG Y,HUANG H,LI C F.Stroke style analysis for painterly rendering [J].Journal of Computer Science and Technology,2013,28(5):762-775.
[6]GATYS L A,Ecker A S,BETHGE M.Image style transfer using convolutional neural networks [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:2414-2423.
[7]GATYS L A,BETHGE M,HERTZMANN A,et al.Preserving color in neural artistic style transfer[J].arXiv:1606.05897,2016.
[8]JOHNSON J,ALAHI A,LI F F.Perceptual losses for real-time style transfer and super-resolution [C]//Proceedings of Euro-pean Conference on Computer Vision.Cham:Springer,2016:694-711.
[9]DUMOULIN V,SHLENS J,KUDLUR M,et al.A learned representation for artistic style [J].arXiv:1610.07629,2016.
[10]SHELHAMER E,LONG J,DARRELL T.Fully convolutional networks for semantic segmentation [J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,39(4):640-651.
[11]LAFFERTY J D,MCCALLUM A,PEREIRA F C N.Condi-tional random fields:probabilistic models for segmenting and labeling sequence data [C]//Proceedings of Eighteenth International Confe-rence on Machine Learning.Morgan Kaufmann Publishers Inc.,2001:282-289.
[12]DENG J,DONG W,SOCHER R,et al.ImageNet:A large-scale hierarchical image database [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2009:710-719.
[13]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition [J].arXiv:1409.1556,2014.
[14]HE K,ZHANG X,REN S,et al.Deep residual learning for ima-ge recognition [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:770-778.
[15]MALKAUTHEKAR M D.Analysis of euclidean distance andManhattan Distance measure in face recognition [C]//International Conference on Computational Intelligence and Information Technology.IET,2013:503-507.
[16]ABADI M,AGARWAL A,BARHAM P,et al.Tensorflow:large-scale machine learning on heterogeneous distributed systems [J].arXiv:1603.04467,2016.
[1] ZHOU Yan, ZENG Fan-zhi, WU Chen, LUO Yue, LIU Zi-qin. 3D Shape Feature Extraction Method Based on Deep Learning [J]. Computer Science, 2019, 46(9): 47-58.
[2] MA Lu, PEI Wei, ZHU Yong-ying, WANG Chun-li, WANG Peng-qian. Fall Action Recognition Based on Deep Learning [J]. Computer Science, 2019, 46(9): 106-112.
[3] LI Qing-hua, LI Cui-ping, ZHANG Jing, CHEN Hong, WANG Shao-qing. Survey of Compressed Deep Neural Network [J]. Computer Science, 2019, 46(9): 1-14.
[4] WANG Yan-ran, CHEN Qing-liang, WU Jun-jun. Research on Image Semantic Segmentation for Complex Environments [J]. Computer Science, 2019, 46(9): 36-46.
[5] SUN Zhong-feng, WANG Jing. RCNN-BGRU-HN Network Model for Aspect-based Sentiment Analysis [J]. Computer Science, 2019, 46(9): 223-228.
[6] DENG Cun-bin, YU Hui-qun, FAN Gui-sheng. Integrating Dynamic Collaborative Filtering and Deep Learning for Recommendation [J]. Computer Science, 2019, 46(8): 28-34.
[7] DU Wei, DING Shi-fei. Overview on Multi-agent Reinforcement Learning [J]. Computer Science, 2019, 46(8): 1-8.
[8] GUO Xu, ZHU Jing-hua. Deep Neural Network Recommendation Model Based on User Vectorization Representation and Attention Mechanism [J]. Computer Science, 2019, 46(8): 111-115.
[9] ZHANG Yi-jie, LI Pei-feng, ZHU Qiao-ming. Event Temporal Relation Classification Method Based on Self-attention Mechanism [J]. Computer Science, 2019, 46(8): 244-248.
[10] LI Zhou-jun,WANG Chang-bao. Survey on Deep-learning-based Machine Reading Comprehension [J]. Computer Science, 2019, 46(7): 7-12.
[11] ZHANG Lin-na,CHEN Jian-qiang,CHEN Xiao-ling,CEN Yi-gang,KAN Shi-chao. Lightweight SSD Network for Real-time Object Detection in Automotive Videos [J]. Computer Science, 2019, 46(7): 233-237.
[12] LI Jian, YANG Xiang-ru, HE Bin. Geometric Features Matching with Deep Learning [J]. Computer Science, 2019, 46(7): 274-279.
[13] LIU Meng-juan,ZENG Gui-chuan,YUE Wei,QIU Li-zhou,WANG Jia-chang. Review on Click-through Rate Prediction Models for Display Advertising [J]. Computer Science, 2019, 46(7): 38-49.
[14] CHEN Si-wen, LIU Yu-jiang, LIU Dong, SU Chen, ZHAO Di, QIAN Lin-xue, ZHANG Pei-heng. AlexNet Model and Adaptive Contrast Enhancement Based UltrasoundImaging Classification [J]. Computer Science, 2019, 46(6A): 146-152.
[15] PENG Jin-xi, SU Yuan-qi, XUE Xiao-rong. SAR Image Feature Retrieval Method Based on Deep Learning and Synchronic Matrix [J]. Computer Science, 2019, 46(6A): 196-199.
Full text



[1] ZHU Hong, LI Qian-mu and LI De-qiang. Facial Multi-landmarks Localization Based on Single Convolution Neural Network[J]. Computer Science, 2018, 45(4): 273 -277, 284 .
[2] WU Jian-xia, YANG Yong-li. Algorithm for Reducing PAPR of FBMC-OQAM System[J]. Computer Science, 2018, 45(6): 89 -95 .
[3] LAI Wen-xing, DENG Zhong-min. Improved NSGA2 Algorithm Based on Dominant Strength[J]. Computer Science, 2018, 45(6): 187 -192 .
[4] SHEN Xia-jiong, ZHANG Jun-tao, HAN Dao-jun. Short-term Traffic Flow Prediction Model Based on Gradient Boosting Regression Tree[J]. Computer Science, 2018, 45(6): 222 -227,264 .
[5] ZHOU Feng, LI Rong-yu. Convolutional Neural Network Model for Text Classification Based on BGRU Pooling[J]. Computer Science, 2018, 45(6): 235 -240 .
[6] WANG Zhen-chao, HOU Huan-huan and LIAN Rui. Geographic Routing Algorithm Based on Location Prediction in WSN[J]. Computer Science, 2018, 45(5): 59 -63 .
[7] TU Yao-feng, LIU Hui, ZHANG Guo-liang and LIU Chun. Key Techniques of a Kind of Distributed Cache Systems and Their Applications[J]. Computer Science, 2018, 45(5): 156 -162 .
[8] LIU Zhuang, CHAI Xiu-juan and CHEN Xi-lin. Application of Two-stream Faster R-CNN in RGB-D Hand Detection[J]. Computer Science, 2018, 45(5): 232 -237 .
[9] SONG Bo ,YAN Ji-xiong, CHEN Ze-hua. Rapid Knowledge Reduction of Large-scale Truth Table Based on Variable Granularity[J]. Computer Science, 2018, 45(7): 167 -171 .
[10] RAO Yuan, LU Shu-min. KAAS-based Service Mechanism of Technology Resources for Collaborative Innovation[J]. Computer Science, 2018, 45(7): 286 -292 .