计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (9): 129-139.doi: 10.11896/jsjkx.230800098

• 计算机图形学&多媒体 • 上一篇    下一篇

艺术美感增强的图像任意风格迁移

李鑫1, 普园媛1,2, 赵征鹏1, 李煜潘1, 徐丹1   

  1. 1 云南大学信息学院 昆明 650500
    2 云南省高校物联网技术及应用重点实验室 昆明 650500
  • 收稿日期:2023-08-16 修回日期:2024-01-09 出版日期:2024-09-15 发布日期:2024-09-10
  • 通讯作者: 赵征鹏(zhpzhao@ynu.edu.cn)
  • 作者简介:(3323163785@qq.com)
  • 基金资助:
    国家自然科学基金(61271361,61761046,62162068);云南省科技厅应用基础研究计划重点项目(202001BB050043);云南省重大科技专项(202302AF080006)

Image Arbitrary Style Transfer via Artistic Aesthetic Enhancement

LI Xin1, PU Yuanyuan1,2, ZHAO Zhengpeng1, LI Yupan1, XU Dan1   

  1. 1 School of Information Science and Engineering,Yunnan University,Kunming 650500,China
    2 University Key Laboratory of Internet of Things Technology and Application in Yunnan Province,Kunming 650500,China
  • Received:2023-08-16 Revised:2024-01-09 Online:2024-09-15 Published:2024-09-10
  • About author:LI Xin,born in 1997,master.His main research interests include image processing and so on.
    ZHAO Zhengpeng,born in 1973,master,associate professor,master supervisor.His research interests include signal and information processing,compu-ter systems and applications.
  • Supported by:
    National Natural Science Foundation of China(61271361,61761046,62162068),Key Project of Applied Basic Research Program of Yunnan Provincial Department of Science and Technology(202001BB050043) and Major Science and Technology Special Project in Yunnan Province(202302AF080006).

PDF (PC)

381

摘要： 目前的研究表明,通用风格迁移取得了显著成功,即能将任意视觉风格迁移到内容图像。然而,在图像任意风格迁移的评价维度中,只考虑语义结构的保留度和风格图案的多样性是不全面的,还应将艺术美感纳入考量范围。现有方法普遍存在艺术美感不自然的问题——表现为风格化图像中会出现不和谐的图案和明显的伪影,很容易与真实的艺术作品区分开来。针对该问题,提出了一种艺术美感增强的图像任意风格迁移方法。首先,设计了一个多尺度艺术美感增强模块,通过提取不同尺度的风格图像特征,改善了风格化图案不和谐的问题;同时,设计了一个美感风格注意力模块,使用通道注意力机制,根据艺术美感特征的全局美感通道分布自适应地匹配并增强相应的风格特征;最后,提出了一个协方差变换融合模块,将增强后的风格特征的二阶统计数据迁移到对应的内容特征上,在很好地保留内容结构的同时实现了美感增强的风格迁移。通过与4种最新的风格迁移方法进行定性比较,同时进行消融实验,分别验证了所提模块与所加损失函数的有效性;在5项定量指标的对比中,有4项取得最优分数。实验结果表明,所提方法可以生成艺术美感更和谐的风格迁移图像。

关键词: 图像风格迁移, 艺术美感, 通道注意力, 协方差变换, 特征融合

Abstract: Current research has shown remarkable success in universal style transfer,which can transfer arbitrary visual styles to content images.However,in the evaluation dimension of arbitrary style transfer of images,it is not comprehensive to only consi-der the retention of semantic structure and the diversity of style patterns,and the artistic aesthetics should also be taken into account.Existing methods generally have the problem of artistic aesthetic unnaturalness,which is manifested in the disharmonious patterns and obvious artifacts in the stylized images which are easy to distinguish from the real paintings.To solve this problem,a novel artistic aesthetic enhancement image arbitrary style transfer(AAEST) approach is proposed.Specifically,first,a multi-scale artistic aesthetic enhancement module is designed to improve the problem of disharmonious patterns by extracting style image features at different scales.At the same time,an aesthetic-style attention module is designed,which uses the channel attention mechanism to adaptively match and enhance style features according to the global aesthetic channel distribution of the aesthetic features.Finally,a covariance transformation fusion module is proposed to transfer the second-order statistics of the enhanced style features to the corresponding content features,so as to achieve aesthetic-enhanced style transfer while preserving the content structure.The effectiveness of the proposed module and the added loss function are verified by qualitative comparison with the latest four style transfer methods and ablation experiments.In the comparison of five quantitative indicators,four achieve optimal scores.Experimental results show that the proposed method can generate more harmonious style transfer images.

Key words: Image style transfer, Artistic aesthetic, Channel attention, Covariance transformation, Feature fusion

中图分类号: 

  • TP391.41
[1]YANG Q X,PU Y Y,ZHAO Z P,et al.W2GAN:ImportanceWeight and Wavelet feature guided Image-to-Image translation under limited data[J].Computers & Graphics,2023,116:115-127.
[2]WANG C,NIE R,CAO J,et al.IGNFusion:An Unsupervised Information Gate Network for Multimodal Medical Image Fusion[J].IEEE Journal of Selected Topics in Signal Processing,2022,16(4):854-868.
[3]GATYS L A,ECKER A S,BETHGE M.Image style transferusing convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:2414-2423.
[4]LI Y J,FANG C,YANG J M,et al.Universal style transfer via feature transforms.[C]//Advances in Neural Information Processing Systems(NeurIPS).2017:386-396.
[5]HUANG X,BELONGIE S.Arbitrary style transfer in real-time with adaptive instance normalization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV).2017:1501-1510.
[6]LI X T,LIU S F,KAUTZ J,et al.Learning Linear Transformations for Fast Image and Video Style Transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2019:3809-3817.
[7]HU Z Y,JIA J,LIU B,et al.Aesthetic-aware image style transfer[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:3320-3329.
[8]AN J,HUANG S Y,SONG Y B,et al.ArtFlow:UnbiasedImage Style Transfer via Reversible Neural Flows[C]//Procee-dings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2021:862-871.
[9]PARK D Y,LEE K H.Arbitrary style transfer with style-attentional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2019:5880-5888.
[10]DENG Y Y,TANG F,DONG W M,et al.Arbitrary style transfer via multi-adaptation network[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:2719-2727.
[11]LUO X,HAN Z,YANG L,et al.Consistent Style Transfer[J].arXiv:2201.02233,2022.
[12]LIU S H,LIN T W,HE D L,et al.Adaattn:Revisit attention mechanism in arbitrary neural style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV).2021:6649-6658.
[13]ZHANG Y L,FANG C,WANG Y L,et al.Multimodal style transfer via graph cuts[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV).2019:5943-5951.
[14]LI C C,CHEN T.Aesthetic visual quality assessment of pain-tings[J].IEEE Journal of selected topics in Signal Processing,2009,3(2):236-252.
[15]JUSTIN J,ALEXANDRE A,LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the European Conference on Computer Vision(ECCV).Springer,2016:694-711.
[16]ULYANOV D,LEBEDEV V,VEDALDI A,et al.Texture Networks:Feed-forward Synthesis of Textures and Stylized Images[C]//International Conference on Machine Learning(ICML).2016:1349-1357.
[17]LI X,PU Y Y,ZHAO Z,et al.Artistic style transfer based on matching content semantics and style features[J].Journal of Graphics,2023,44(4):699-709.
[18]WANG Z Z,ZHANG Z J,ZHAO L,et al.AesUST:TowardsAesthetic-Enhanced Universal Style Transfer[C]//Proceedings of the 30th ACM International Conference on Multimedia.2022:1095-1106.
[19]WU Z J,ZHU Z,DU J P,et al.CCPL:Contrastive Coherence Preserving Loss for Versatile Style Transfer[C]//European Conference on Computer Vision(ECCV).Lecture Notes in Computer Science,2022.
[20]ZHANG Y X,TANG F,DONG W M,et al.Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning[C]//ACM SIGGRAPH 2022 Conference Proceedings(SIGGRAPH'22).New York:Association for Computing Machinery,2022.
[21]WEN L F,GAO C Y,ZOU C Q.CAP-VSTNet:Content Affinity Preserved Versatile Style Transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2023:18300-18309.
[22]LIC,WAND M.Combining markov random fields and convolutional neural networks for image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2016:2479-2486.
[23]CHEN T Q,SCHMIDTM.Fast patch-based style transfer of arbitrary style[J].arXiv:1612.04337,2016.
[24]SHENG L,LIN Z Y,SHAO J,et al.Avatar-Net:Multi-scale Zero-shot Style Transfer by Feature Decoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2018:8242-8250.
[25]DENG Y Y,TANG F,DONG W M,et al.Arbitrary style transfer via multi-adaptation network[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:2719-2727.
[26]LIU H,PU Y Y,LV D H,et al.Automatic coloring of images with polarized self-attention-constrained color overflow [J].Computer Science,2023,50(3):208-215.
[27]ZHU P J,PU Y Y,ZHAO Z P,et al.A review of the research on natural image coloring [J].Journal of Yunnan University(Natural Science Edition),2023,45(2):314-325.
[28]KONG F M,PU Y Y,ZHAO Z P,et al.Portrait ManHuaHua based on double circular mapping networks [J/OL].Computer Application Research:1-7.[2023-09-27].https://doi.org/10.19734/j.issn.1001-3695.2023.05.0226.
[29]YAO W J,ZHAO Z P,PU Y Y,et al.A Cuan-style migration model for dense adaptive generative adversarial networks [J].Journal of Computer-Aided Design and Graphics,2023,35(6):915-924.
[30]LV W R,PU Y Y,ZHAO Z P et al.Font migration based on font perception and attention normalization [J].Computer Science,2023,50(S1):408-413.
[31] AYDIN T O,SMOLIC A,GROSS M.Automated aesthetic ana-lysis of photographic images[J].IEEE Transactions on Visua-lization and Computer Graphics(TVCG),2014,21(1):31-42.
[32]WANG Z H,ZHAO L,CHEN H B,et al.Evaluate and improve the quality of neural style transfer[J] Computer Vision and Image Understanding,2021(207):103203.
[33]CHEN H B,ZHAO L,ZHANG H M,et al.Diverse image style transfer via invertible cross-space mapping[C] // 2021 IEEE/CVF International Conference on Computer Vision(ICCV).2021:14860-14869.
[34]ZUO Z W,ZHAO L,LIAN S B,et al.Style Fader Generative Adversarial Networks for Style Degree Controllable Artistic Style Transfer[C]//Proceedings of the Thirty-First Interna-tional Joint Conference on Artificial Intelligence(IJCAI).2022:5002-5009.
[35]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition [C]//International Conference on Learning Representations.2015.
[36]WANG T,LIU M Y,ZHU J Y,et al.High-Resolution ImageSynthesis and Semantic Manipulation with Conditional GANs [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.2018:8798-8807.
[37]FU J,LIU J,TIAN H J,et al.Dual attention network for scene segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2019:3146-3154.
[38]DENG Y,TANG F,DONG W,et al.Arbitrary video style transfer via multi-channel correlation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2021:1210-1217.
[39]LIN T Y,MAIRE M,BELONGIE S,et al.Microsoft COCO:Common Objects in Context[C]//Proceedings of the European Conference on Computer Vision(ECCV).Zurich:Springer,2014:740-755.
[40]PHILLIPS F,MACKINTOSH B.WikiArt Gallery,Inc.:A case for critical thinking[J].Issues in Accounting Education,2011,26(3):593-608.
[41]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[42]HEUSEL M,RAMSAUER H,UNTERTHINER T,et al.Gans trained by a two time-scale update rule converge to a local nash equilibrium[C]//Advances in Neural Information Processing Systems.2017.
[43]WANG Z Z,ZHAO L,CHEN H B,et al.Evaluate and improve the quality of neural style transfer[J].Computer Vision and Image Understanding,2021,207:103203.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发