Computer Science ›› 2022, Vol. 49 ›› Issue (7): 100-105.doi: 10.11896/jsjkx.210600036

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Photorealistic Style Transfer Guided by Global Information

ZHANG Ying-tao, ZHANG Jie, ZHANG Rui, ZHANG Wen-qiang   

  1. School of Computer Science,Fudan University,Shanghai 200011,China
    Shanghai Key Laboratory of Intelligent Information Processing,Fudan University,Shanghai 200011,China
  • Received:2021-06-03 Revised:2021-10-17 Online:2022-07-15 Published:2022-07-12
  • About author:ZHANG Ying-tao,born in 1997,postgraduate.His main research interests include computer vision,deep learning and machine learning.
    ZHANG Rui,born in 1973,Ph.D,senior engineer.His main research interests include embedded system,digital signal process and mobile communications.

Abstract: Different from artistic style transfer,the challenge of photorealistic style transfer is to maintain the authenticity of the output while transferring the color style of the style input.Now,most photorealistic style transfer methods perform pre-proces-sing or post-processing based on artistic style transfer methods,to maintain the authenticity of the output image.However,artistic style transfer methods usually cannot make full use of global color information to achieve a more coordinated overall impression,and pre-processing and post-processing operations are often tedious and time-consuming.To solve the above problems,this paper establishes a photorealistic style transfer network guided by global information,and proposes a color-partition-mean loss(Lcpm) to measure the similarity of the global color distribution between output and the style input.Adaptive instance normalization(AdaIN) is improved,and partition adaptive instance normalization(AdaIN-P) is proposed to better adapt to the color style transfer of real images.In addition,this paper also introduces a cross-channel partition attention module to make better use of global context information and improve the overall coordination of output images.Through the above methods,the decoder of network is guided to make full use of global information.Experimental results show that,compared with other state-of-the-art me-thods,the proposed model can achieve a better photorealistic style transfer effect while maintaining image details.

Key words: Attention mechanism, Convolution neural network, Encoder and decoder, Feature fusion, Global information, Style transfer

CLC Number: 

  • TP391
[1]GATYS L A,ECKER A S,Bethge M.Texture Synthesis Using Convolutional Neural Networks[C]//Advances in Neural Information Processing Systems.2015:262-270.
[2]GATYS L A,ECKER A S,BETHGE M.Image style transferusing convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.NJ:IEEE,2016:2414-2423.
[3]LUAN F,PARIS S,SHECHTMAN E,et al.Deep Photo Style Transfer[C]//Proceedings of the IEEE Conference on Compu-ter Vision and Pattern Recognition.NJ:IEEE,2017:4990-4998.
[4]LI Y,LIU M,LI X,et al.A closed-form solution to photorealistic image stylization[C]//Proceedings of the European Confe-rence on Computer vision.Berlin:Springer,2018:453-468.
[5]YOO J,UH Y,CHUN S,et al.Photorealistic Style Transfer via Wavelet Transforms[C]//Proceedings of the IEEE InternationalConference on Computer Vision.NJ:IEEE,2019:9036-9045.
[6]AN J,XIONG H,HUAN J,et al.Ultrafast Photorealistic Style Transfer via Neural Architecture Search[C]//AAAI Conference on Artificial Intelligence.2020:10443-10450.
[7]HUANG X,BELONGIE S.Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization[C]//Proceedings of the IEEE International Conference on Computer Vision.NJ:IEEE,2017:1501-1510.
[8]LI Y,FANG C,YANG J,et al.Universal style transfer via feature transforms[C]//Advances in Neural Information Proces-sing Systems.2017:386-396.
[9]JOHNSON J,ALAHI A,FEI-FEI L.Perceptual Losses forReal-Time Style Transfer and Super-Resolution[C]//Procee-dings of the European Conference on Computer Vision.Berlin:Springer,2016:694-711.
[10]CHEN D,YUAN L,LIAO J,et al.StyleBank:An Explicit Representation for Neural Image Style Transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE International Conference on Computer Vision.NJ:IEEE,2017:1897-1906.
[11]ULYANOV D,VEDALDI A,LEMPITSKY V.Instance nor-malization:The missing ingredient for fast stylization[J].ar-Xiv:1607.08022,2016.
[12]GHIASI G,LEE H,KUDLUR M,et al.Exploring the structure of a real-time,arbitrary neural artistic stylization network[J].arXiv:1705.06830,2017.
[13]HERTZMAN A,JACOBS C E,OLIVER N,et al.Image analogies[C]//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques.2001:327-340.
[14]ASHIKHMIN N.Fast Texture Transfer[J].IEEE Computer Graphics & Applications,2003,23(4):38-43.
[15]ULYANOV D,VEDALDI A,LEMPITSKY V.Improved texture networks:Maximizing quality and diversity in feed-forward stylization and texture synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6924-6932.
[16]REINHARD E,ASHIKHMIN M,GOOCH B,et al.ColorTransfer between Images[J].IEEE Computer Graphics and Applications,2001,21(5):34-41.
[17]WELSH T,ASHIKHMIN M,MUELLER K.Transferring color to greyscale images[C]//Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques.New York:ACM,2002:277-280.
[18]ZOPH B,LE Q V.Neural architecture search with reinforcement learning[J].arXiv:1611.01578,2016.
[19]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[20]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation [C]//International Conference on Medical Image Computing and Compu-ter-assisted Intervention.Berlin:Springer,2015:234-241.
[21]HUANG Z,WANG X,HUANG L,et al.Ccnet:Criss-cross attention for semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision.NJ:IEEE,2019:603-612.
[22]HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.NJ:IEEE,2018:7132-7141.
[23]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[1] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[2] ZHOU Fang-quan, CHENG Wei-qing. Sequence Recommendation Based on Global Enhanced Graph Neural Network [J]. Computer Science, 2022, 49(9): 55-63.
[3] DAI Yu, XU Lin-feng. Cross-image Text Reading Method Based on Text Line Matching [J]. Computer Science, 2022, 49(9): 139-145.
[4] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[5] XIONG Li-qin, CAO Lei, LAI Jun, CHEN Xi-liang. Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization [J]. Computer Science, 2022, 49(9): 172-182.
[6] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[7] WANG Ming, PENG Jian, HUANG Fei-hu. Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction [J]. Computer Science, 2022, 49(8): 40-48.
[8] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[9] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[10] YAN Jia-dan, JIA Cai-yan. Text Classification Method Based on Information Fusion of Dual-graph Neural Network [J]. Computer Science, 2022, 49(8): 230-236.
[11] JIN Fang-yan, WANG Xiu-li. Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM [J]. Computer Science, 2022, 49(7): 179-186.
[12] XIONG Luo-geng, ZHENG Shang, ZOU Hai-tao, YU Hua-long, GAO Shang. Software Self-admitted Technical Debt Identification with Bidirectional Gate Recurrent Unit and Attention Mechanism [J]. Computer Science, 2022, 49(7): 212-219.
[13] PENG Shuang, WU Jiang-jiang, CHEN Hao, DU Chun, LI Jun. Satellite Onboard Observation Task Planning Based on Attention Neural Network [J]. Computer Science, 2022, 49(7): 242-247.
[14] ZENG Zhi-xian, CAO Jian-jun, WENG Nian-feng, JIANG Guo-quan, XU Bin. Fine-grained Semantic Association Video-Text Cross-modal Entity Resolution Based on Attention Mechanism [J]. Computer Science, 2022, 49(7): 106-112.
[15] CHENG Cheng, JIANG Ai-lian. Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction [J]. Computer Science, 2022, 49(7): 120-126.
Full text



No Suggested Reading articles found!