Computer Science

Computer Science ›› 2024, Vol. 51 ›› Issue (9): 129-139.doi: 10.11896/jsjkx.230800098

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Image Arbitrary Style Transfer via Artistic Aesthetic Enhancement

LI Xin1, PU Yuanyuan1,2, ZHAO Zhengpeng1, LI Yupan1, XU Dan1   

  1. 1 School of Information Science and Engineering,Yunnan University,Kunming 650500,China
    2 University Key Laboratory of Internet of Things Technology and Application in Yunnan Province,Kunming 650500,China
  • Received:2023-08-16 Revised:2024-01-09 Online:2024-09-15 Published:2024-09-10
  • About author:LI Xin,born in 1997,master.His main research interests include image processing and so on.
    ZHAO Zhengpeng,born in 1973,master,associate professor,master supervisor.His research interests include signal and information processing,compu-ter systems and applications.
  • Supported by:
    National Natural Science Foundation of China(61271361,61761046,62162068),Key Project of Applied Basic Research Program of Yunnan Provincial Department of Science and Technology(202001BB050043) and Major Science and Technology Special Project in Yunnan Province(202302AF080006).

PDF (PC)

377

Abstract: Current research has shown remarkable success in universal style transfer,which can transfer arbitrary visual styles to content images.However,in the evaluation dimension of arbitrary style transfer of images,it is not comprehensive to only consi-der the retention of semantic structure and the diversity of style patterns,and the artistic aesthetics should also be taken into account.Existing methods generally have the problem of artistic aesthetic unnaturalness,which is manifested in the disharmonious patterns and obvious artifacts in the stylized images which are easy to distinguish from the real paintings.To solve this problem,a novel artistic aesthetic enhancement image arbitrary style transfer(AAEST) approach is proposed.Specifically,first,a multi-scale artistic aesthetic enhancement module is designed to improve the problem of disharmonious patterns by extracting style image features at different scales.At the same time,an aesthetic-style attention module is designed,which uses the channel attention mechanism to adaptively match and enhance style features according to the global aesthetic channel distribution of the aesthetic features.Finally,a covariance transformation fusion module is proposed to transfer the second-order statistics of the enhanced style features to the corresponding content features,so as to achieve aesthetic-enhanced style transfer while preserving the content structure.The effectiveness of the proposed module and the added loss function are verified by qualitative comparison with the latest four style transfer methods and ablation experiments.In the comparison of five quantitative indicators,four achieve optimal scores.Experimental results show that the proposed method can generate more harmonious style transfer images.

Key words: Image style transfer, Artistic aesthetic, Channel attention, Covariance transformation, Feature fusion

CLC Number: 

  • TP391.41
[1]YANG Q X,PU Y Y,ZHAO Z P,et al.W2GAN:ImportanceWeight and Wavelet feature guided Image-to-Image translation under limited data[J].Computers & Graphics,2023,116:115-127.
[2]WANG C,NIE R,CAO J,et al.IGNFusion:An Unsupervised Information Gate Network for Multimodal Medical Image Fusion[J].IEEE Journal of Selected Topics in Signal Processing,2022,16(4):854-868.
[3]GATYS L A,ECKER A S,BETHGE M.Image style transferusing convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:2414-2423.
[4]LI Y J,FANG C,YANG J M,et al.Universal style transfer via feature transforms.[C]//Advances in Neural Information Processing Systems(NeurIPS).2017:386-396.
[5]HUANG X,BELONGIE S.Arbitrary style transfer in real-time with adaptive instance normalization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV).2017:1501-1510.
[6]LI X T,LIU S F,KAUTZ J,et al.Learning Linear Transformations for Fast Image and Video Style Transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2019:3809-3817.
[7]HU Z Y,JIA J,LIU B,et al.Aesthetic-aware image style transfer[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:3320-3329.
[8]AN J,HUANG S Y,SONG Y B,et al.ArtFlow:UnbiasedImage Style Transfer via Reversible Neural Flows[C]//Procee-dings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2021:862-871.
[9]PARK D Y,LEE K H.Arbitrary style transfer with style-attentional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2019:5880-5888.
[10]DENG Y Y,TANG F,DONG W M,et al.Arbitrary style transfer via multi-adaptation network[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:2719-2727.
[11]LUO X,HAN Z,YANG L,et al.Consistent Style Transfer[J].arXiv:2201.02233,2022.
[12]LIU S H,LIN T W,HE D L,et al.Adaattn:Revisit attention mechanism in arbitrary neural style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV).2021:6649-6658.
[13]ZHANG Y L,FANG C,WANG Y L,et al.Multimodal style transfer via graph cuts[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV).2019:5943-5951.
[14]LI C C,CHEN T.Aesthetic visual quality assessment of pain-tings[J].IEEE Journal of selected topics in Signal Processing,2009,3(2):236-252.
[15]JUSTIN J,ALEXANDRE A,LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the European Conference on Computer Vision(ECCV).Springer,2016:694-711.
[16]ULYANOV D,LEBEDEV V,VEDALDI A,et al.Texture Networks:Feed-forward Synthesis of Textures and Stylized Images[C]//International Conference on Machine Learning(ICML).2016:1349-1357.
[17]LI X,PU Y Y,ZHAO Z,et al.Artistic style transfer based on matching content semantics and style features[J].Journal of Graphics,2023,44(4):699-709.
[18]WANG Z Z,ZHANG Z J,ZHAO L,et al.AesUST:TowardsAesthetic-Enhanced Universal Style Transfer[C]//Proceedings of the 30th ACM International Conference on Multimedia.2022:1095-1106.
[19]WU Z J,ZHU Z,DU J P,et al.CCPL:Contrastive Coherence Preserving Loss for Versatile Style Transfer[C]//European Conference on Computer Vision(ECCV).Lecture Notes in Computer Science,2022.
[20]ZHANG Y X,TANG F,DONG W M,et al.Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning[C]//ACM SIGGRAPH 2022 Conference Proceedings(SIGGRAPH'22).New York:Association for Computing Machinery,2022.
[21]WEN L F,GAO C Y,ZOU C Q.CAP-VSTNet:Content Affinity Preserved Versatile Style Transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2023:18300-18309.
[22]LIC,WAND M.Combining markov random fields and convolutional neural networks for image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2016:2479-2486.
[23]CHEN T Q,SCHMIDTM.Fast patch-based style transfer of arbitrary style[J].arXiv:1612.04337,2016.
[24]SHENG L,LIN Z Y,SHAO J,et al.Avatar-Net:Multi-scale Zero-shot Style Transfer by Feature Decoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2018:8242-8250.
[25]DENG Y Y,TANG F,DONG W M,et al.Arbitrary style transfer via multi-adaptation network[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:2719-2727.
[26]LIU H,PU Y Y,LV D H,et al.Automatic coloring of images with polarized self-attention-constrained color overflow [J].Computer Science,2023,50(3):208-215.
[27]ZHU P J,PU Y Y,ZHAO Z P,et al.A review of the research on natural image coloring [J].Journal of Yunnan University(Natural Science Edition),2023,45(2):314-325.
[28]KONG F M,PU Y Y,ZHAO Z P,et al.Portrait ManHuaHua based on double circular mapping networks [J/OL].Computer Application Research:1-7.[2023-09-27].https://doi.org/10.19734/j.issn.1001-3695.2023.05.0226.
[29]YAO W J,ZHAO Z P,PU Y Y,et al.A Cuan-style migration model for dense adaptive generative adversarial networks [J].Journal of Computer-Aided Design and Graphics,2023,35(6):915-924.
[30]LV W R,PU Y Y,ZHAO Z P et al.Font migration based on font perception and attention normalization [J].Computer Science,2023,50(S1):408-413.
[31] AYDIN T O,SMOLIC A,GROSS M.Automated aesthetic ana-lysis of photographic images[J].IEEE Transactions on Visua-lization and Computer Graphics(TVCG),2014,21(1):31-42.
[32]WANG Z H,ZHAO L,CHEN H B,et al.Evaluate and improve the quality of neural style transfer[J] Computer Vision and Image Understanding,2021(207):103203.
[33]CHEN H B,ZHAO L,ZHANG H M,et al.Diverse image style transfer via invertible cross-space mapping[C] // 2021 IEEE/CVF International Conference on Computer Vision(ICCV).2021:14860-14869.
[34]ZUO Z W,ZHAO L,LIAN S B,et al.Style Fader Generative Adversarial Networks for Style Degree Controllable Artistic Style Transfer[C]//Proceedings of the Thirty-First Interna-tional Joint Conference on Artificial Intelligence(IJCAI).2022:5002-5009.
[35]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition [C]//International Conference on Learning Representations.2015.
[36]WANG T,LIU M Y,ZHU J Y,et al.High-Resolution ImageSynthesis and Semantic Manipulation with Conditional GANs [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.2018:8798-8807.
[37]FU J,LIU J,TIAN H J,et al.Dual attention network for scene segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2019:3146-3154.
[38]DENG Y,TANG F,DONG W,et al.Arbitrary video style transfer via multi-channel correlation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2021:1210-1217.
[39]LIN T Y,MAIRE M,BELONGIE S,et al.Microsoft COCO:Common Objects in Context[C]//Proceedings of the European Conference on Computer Vision(ECCV).Zurich:Springer,2014:740-755.
[40]PHILLIPS F,MACKINTOSH B.WikiArt Gallery,Inc.:A case for critical thinking[J].Issues in Accounting Education,2011,26(3):593-608.
[41]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[42]HEUSEL M,RAMSAUER H,UNTERTHINER T,et al.Gans trained by a two time-scale update rule converge to a local nash equilibrium[C]//Advances in Neural Information Processing Systems.2017.
[43]WANG Z Z,ZHAO L,CHEN H B,et al.Evaluate and improve the quality of neural style transfer[J].Computer Vision and Image Understanding,2021,207:103203.
[1] LIU Qian, BAI Zhihao, CHENG Chunling, GUI Yaocheng. Image-Text Sentiment Classification Model Based on Multi-scale Cross-modal Feature Fusion [J]. Computer Science, 2024, 51(9): 258-264.
[2] LIU Sichun, WANG Xiaoping, PEI Xilong, LUO Hangyu. Scene Segmentation Model Based on Dual Learning [J]. Computer Science, 2024, 51(8): 133-142.
[3] WANG Chao, TANG Chao, WANG Wenjian, ZHANG Jing. Infrared Human Action Recognition Method Based on Multimodal Attention Network [J]. Computer Science, 2024, 51(8): 232-241.
[4] CAI Wenliang, HUANG Jun. Lane Detection Method Based on RepVGG [J]. Computer Science, 2024, 51(7): 236-243.
[5] WANG Yanlin, SUN Jing, YANG Hongbo, GUO Tao, PAN Jiahua, WANG Weilian. Classification Model of Heart Sounds in Pulmonary Hypertension Based on Time-Frequency Fusion Features [J]. Computer Science, 2024, 51(6A): 230800091-7.
[6] KANG Zhiyong, LI Bicheng, LIN Huang. User Interest Recognition Method Incorporating Category Labels and Topic Information [J]. Computer Science, 2024, 51(6A): 230500169-8.
[7] HAN Zhigeng, ZHOU Ting, CHEN Geng, FU Chunshuo, CHEN Jian. RM-RT2NI:A Recommendation Model with Review Timeliness and Trusted Neighbor Influence [J]. Computer Science, 2024, 51(6A): 230800160-7.
[8] LI Guo, CHEN Chen, YANG Jing, QUN Nuo. Study on Tibetan Short Text Classification Based on DAN and FastText [J]. Computer Science, 2024, 51(6A): 230700064-5.
[9] QUE Yue, GAN Menghan, LIU Zhiwei. Object Detection with Receptive Field Expansion and Multi-branch Aggregation [J]. Computer Science, 2024, 51(6A): 230600151-6.
[10] LIU Heng, LIN Hongyu, WU Tao. Detection Method for Workers’ Illegal Operation Behavior in PackagingWorkshop of CigaretteFactory [J]. Computer Science, 2024, 51(6A): 230700123-8.
[11] LI Yuehao, WANG Dengjiang, JIAN Haifang, WANG Hongchang, CHENG Qinghua. LiDAR-Radar Fusion Object Detection Algorithm Based on BEV Occupancy Prediction [J]. Computer Science, 2024, 51(6): 215-222.
[12] GAO Nan, ZHANG Lei, LIANG Ronghua, CHEN Peng, FU Zheng. Scene Text Detection Algorithm Based on Feature Enhancement [J]. Computer Science, 2024, 51(6): 256-263.
[13] HONG Tijing, LIU Dengfeng, LIU Yian. Radar Active Jamming Recognition Based on Multiscale Fully Convolutional Neural Network and GRU [J]. Computer Science, 2024, 51(5): 306-312.
[14] SHAN Xinxin, LI Kai, WEN Ying. Medical Image Segmentation Network Integrating Full-scale Feature Fusion and RNN with Attention [J]. Computer Science, 2024, 51(5): 100-107.
[15] ZHOU Yu, CHEN Zhihua, SHENG Bin, LIANG Lei. Multi Scale Progressive Transformer for Image Dehazing [J]. Computer Science, 2024, 51(5): 117-124.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.