基于卷积神经网络的图像局部风格迁移

doi:10.11896/j.issn.1002-137X.2019.09.039

计算机科学 ›› 2019, Vol. 46 ›› Issue (9): 259-264.doi: 10.11896/j.issn.1002-137X.2019.09.039

基于卷积神经网络的图像局部风格迁移

缪永伟^1,2, 李高怡¹, 鲍陈¹, 张旭东¹, 彭思龙³

(浙江工业大学计算机科学与技术学院杭州310023)¹;
(浙江理工大学信息学院杭州310018)²;
(中国科学院自动化研究所北京100190)³

收稿日期:2018-07-20 出版日期:2019-09-15 发布日期:2019-09-02
通讯作者: 缪永伟(1971－),男,博士,教授,博士生导师,CCF高级会员,主要研究方向为计算机图形学、数字几何处理、计算机视觉、机器学习,E-mail:ywmiao@zstu.edu.cn
作者简介:李高怡(1994－),女,硕士生,主要研究方向为计算机视觉、机器学习;鲍陈(1983－),男,博士生,主要研究方向为计算机图形学、计算机视觉、机器学习;张旭东(1982－),男,博士,讲师,主要研究方向为计算机图形学、计算机视觉、机器学习;彭思龙(1971－),男,博士,研究员,博士生导师,主要研究方向为计算机视觉、机器学习。
基金资助:
国家自然科学基金项目(61272309)

Image Localized Style Transfer Based on Convolutional Neural Network

MIAO Yong-wei^1,2, LI Gao-yi¹, BAO Chen¹, ZHANG Xu-dong¹, PENG Si-long³

(College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China)¹;
(College of Information Science and Technology,Zhejiang Sci-Tech University,Hangzhou 310018,China)²;
(Institute of Automation,Chinese Academy of Sciences,Beijing 100190,China)³

Received:2018-07-20 Online:2019-09-15 Published:2019-09-02

摘要/Abstract

摘要： 图像风格迁移是计算机图形学和计算机视觉的一个研究热点。针对现有的图像风格迁移方法中难以对内容图局部区域进行风格迁移的难点,提出了一种基于卷积神经网络的图像局部风格迁移框架。首先,根据输入的内容图和风格图,利用图像风格迁移网络生成全局风格迁移图;然后,利用图像语义分割网络,通过自动语义分割生成的掩码确定图像前景区域与背景区域;最后,利用掩码图确定风格迁移区域并融合未迁移区域得到图像局部风格迁移结果,同时提出一种基于曼哈顿距离的图像融合算法以优化局部风格迁移对象与未迁移区域之间边界的衔接和平滑过渡。该框架综合考虑了目标区域和边界带的像素值、位置等细节信息,在3个公开的图像数据集上进行实验,结果表明该方法能够高效、快速并自然地实现输入内容图的局部风格迁移,生成艺术性与真实性和谐并存的视觉效果。

关键词: 卷积神经网络, 曼哈顿距离, 深度学习, 图像局部风格迁移, 自动语义分割

Abstract: Image style transfer is a research hot topic in computer graphics and computer vision.Aiming at the difficulty in the style transfer of the local area of the content image in the existing image style transfer method,this paper proposed a localized image transfer framework based on convolutional neural network.First,according to the input content image and style image,the image style transfer network is used to generate the whole style transferred image.Then,the image foreground and the background area are determined by the mask generated by automatic semantic segmentation.Finally,according to style transfer result of the foreground or the background region,an image fusion algorithm based on Manhattan distance is proposed to optimize the convergence and smooth transition between the stylized object and the original area.The framework comprehensively considers the pixel values and positions of the target area and the boundary band,and experiments on three public image datasets demonstrate that the method can efficiently,quickly and naturally implement local style transfer of input content maps,and produce visual effects that are both artistic and authentic.

Key words: Automatic semantic segmentation, Convolutional neural network (CNN), Deep learning, Localized image style transfer, Manhattan distance

中图分类号:

TP391

缪永伟, 李高怡, 鲍陈, 张旭东, 彭思龙. 基于卷积神经网络的图像局部风格迁移[J]. 计算机科学, 2019, 46(9): 259-264. https://doi.org/10.11896/j.issn.1002-137X.2019.09.039

MIAO Yong-wei, LI Gao-yi, BAO Chen, ZHANG Xu-dong, PENG Si-long. Image Localized Style Transfer Based on Convolutional Neural Network[J]. Computer Science, 2019, 46(9): 259-264. https://doi.org/10.11896/j.issn.1002-137X.2019.09.039

参考文献

[1]GOOCH B,GOOCH A.Non-photorealistic rendering [M].New York:AK Peters/CRC Press,2001.
[2]DECARLO D,SANTELL A.Stylization and abstraction of photographs [J].ACM Transactions on Graphics,2002,21(3):769-776.
[3]WANG J,XU Y,SHUM H Y,et al.Video toning [J].ACM Transactions on Graphics,2004,23(3):574-583.
[4]LI P,SUN H Q,SHENG B,et al.Image stylization with enhanced structure on GPU [J].Science China Information Scie-nces,2012,55(5):1093-1105.
[5]ZANG Y,HUANG H,LI C F.Stroke style analysis for painterly rendering [J].Journal of Computer Science and Technology,2013,28(5):762-775.
[6]GATYS L A,Ecker A S,BETHGE M.Image style transfer using convolutional neural networks [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:2414-2423.
[7]GATYS L A,BETHGE M,HERTZMANN A,et al.Preserving color in neural artistic style transfer[J].arXiv:1606.05897,2016.
[8]JOHNSON J,ALAHI A,LI F F.Perceptual losses for real-time style transfer and super-resolution [C]//Proceedings of Euro-pean Conference on Computer Vision.Cham:Springer,2016:694-711.
[9]DUMOULIN V,SHLENS J,KUDLUR M,et al.A learned representation for artistic style [J].arXiv:1610.07629,2016.
[10]SHELHAMER E,LONG J,DARRELL T.Fully convolutional networks for semantic segmentation [J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,39(4):640-651.
[11]LAFFERTY J D,MCCALLUM A,PEREIRA F C N.Condi-tional random fields:probabilistic models for segmenting and labeling sequence data [C]//Proceedings of Eighteenth International Confe-rence on Machine Learning.Morgan Kaufmann Publishers Inc.,2001:282-289.
[12]DENG J,DONG W,SOCHER R,et al.ImageNet:A large-scale hierarchical image database [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2009:710-719.
[13]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition [J].arXiv:1409.1556,2014.
[14]HE K,ZHANG X,REN S,et al.Deep residual learning for ima-ge recognition [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:770-778.
[15]MALKAUTHEKAR M D.Analysis of euclidean distance andManhattan Distance measure in face recognition [C]//International Conference on Computational Intelligence and Information Technology.IET,2013:503-507.
[16]ABADI M,AGARWAL A,BARHAM P,et al.Tensorflow:large-scale machine learning on heterogeneous distributed systems [J].arXiv:1603.04467,2016.

相关文章 15

[1]	周乐员, 张剑华, 袁甜甜, 陈胜勇. 多层注意力机制融合的序列到序列中国连续手语识别和翻译 Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion 计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[2]	徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[3]	饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[4]	汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[5]	陈泳全, 姜瑛. 基于卷积神经网络的APP用户行为分析方法 Analysis Method of APP User Behavior Based on Convolutional Neural Network 计算机科学, 2022, 49(8): 78-85. https://doi.org/10.11896/jsjkx.210700121
[6]	朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥. 基于注意力机制的医学影像深度哈希检索算法 Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism 计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[7]	孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[8]	檀莹莹, 王俊丽, 张超波. 基于图卷积神经网络的文本分类方法研究综述 Review of Text Classification Methods Based on Graph Convolutional Network 计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064
[9]	李宗民, 张玉鹏, 刘玉杰, 李华. 基于可变形图卷积的点云表征学习 Deformable Graph Convolutional Networks Based Point Cloud Representation Learning 计算机科学, 2022, 49(8): 273-278. https://doi.org/10.11896/jsjkx.210900023
[10]	王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[11]	郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[12]	姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[13]	侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木. 中文预训练模型研究进展 Advances in Chinese Pre-training Models 计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[14]	周慧, 施皓晨, 屠要峰, 黄圣君. 基于主动采样的深度鲁棒神经网络学习 Robust Deep Neural Network Learning Based on Active Sampling 计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[15]	金方焱, 王秀利. 融合RACNN和BiLSTM的金融领域事件隐式因果关系抽取 Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM 计算机科学, 2022, 49(7): 179-186. https://doi.org/10.11896/jsjkx.210500190

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于卷积神经网络的图像局部风格迁移

Image Localized Style Transfer Based on Convolutional Neural Network

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

Metrics

本文评价

推荐阅读 0