Computer Science

Computer Science ›› 2020, Vol. 47 ›› Issue (9): 142-149.doi: 10.11896/jsjkx.190900203

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Expression Animation Synthesis Based on Improved CycleGan Model and Region Segmentation

YE Ya-nan1,2, CHI Jing1,2, YU Zhi-ping1,2, ZHAN Yu-li1,2and ZHANG Cai-ming1,2,3,4   

  1. 1 School of Computer Science and Technology,Shandong University of Finance and Economics,Jinan 250014,China
    2 Shandong Provincial Key Laboratory of Digital Media Technology,Jinan 250014,China
    3 School of Software,Shandong University,Jinan 250101,China
    4 Future Intelligent Computing Collaborative Innovation Center,Yantai,Shandong 264003,China
  • Received:2019-06-16 Published:2020-09-10
  • About author:YE Ya-nan,born in 1994,master,postgraduate.Her main research interests include computer animation and digital image processing.
    CHI Jing,born in 1980,Ph.D,associate professor,postgraduate supervisor.Her main research interests includecompu-ter animation,geometric shape,and me-dical image processing.
  • Supported by:
    Natural Science Foundation of Shandong Province for Excellent Young Scholars in Provincial Universities (ZR2018JL022),National Natural Science Foundation of China (61772309,61602273),Shandong Provincial Key R&D Program (2019GSF109112),Science and Technology Program of Shandong Education Department (J18RA272) and Fostering Project of Dominant Discipline and Talent Team of Shandong Province Higher Education Institutions.

PDF (PC)

1375

Abstract: Aiming at the problems of mostly relying on data source driver,low generation efficiency and poor authenticity of the existing facial expression synthesis methods,this paper proposes a new method for expression animation synthesis based on the improved CycleGan model and region segmentation.This new method can synthesize new expression in real time and has good stability and robustness.The proposed method constructs a new covariance constraint in the cycle consistent loss function of the traditional CycleGan model,which can effectively avoid color anomalies and image blurring in generation of new expression images.The idea of zonal training is put forward.The Dlib face recognition database is used to detect the key points of the face images.The detected key feature points are used to segment the face in domain source and target domain into four zones:left eye,right eye,mouth and the rest of the face.The improved CycleGan model is used to train each region separately,and finally the training results are weighted and fused into the final new expression image.The zonal training further enhances the authenticity of expression synthesis.The experimental data comes from the SAVEE database,and the experimental results are presented with python 3.4 software under the Tensorflow framework.Experiments show that the new method can directly generate real and natu-ral new expression sequences in real time on the original facial expression sequence without data source driver.Furthermore,for the voice video,it can effectively ensure the synchronization between the generated facial expression sequence and the source audio.

Key words: Covariance constraint, CycleGan, Deep learning, Facial expression synthesis, Region segmentation

CLC Number: 

  • TP391.41
[1] ZHU J,PARK T,ISOLA P,et al.Unpaired Image-to-ImageTranslation Using Cycle-Consistent Adversarial Networks[C]//2017 IEEE International Conference on Computer Vision (ICCV).2017:2242-2251.
[2] PIGHIN F,HECKER J,LISCHINSKI D,et al.Synthesizing Realistic Facial Expressions from Photographs[C]//Proceedings of the ACM SIGGRAPH Conference on Computer Graphics.1998:75-84.
[3] BLANZ V,BASSO C,VETEER T,et al.Reanimating Faces in Images and Video[C]//European Association for Computer Graphics.2003:641-650.
[4] VLASIC D,BRAND M,PFISTER H,et al.Face Transfer with Multilinear Models[J].ACM Transactions on Graphics,2006,24(3):426-433.
[5] LV P,XU M L.Expression of Face Expre-ssions Unrelated to Expression Database[J].Journal of Computer-Aided Design & Computer Graphics,2016,28(1).
[6] PASQUARIELLO S,PELACHAUD C.GRETA:A Simple Facial Animation Engine[M]//Soft Computing and Industry.London:Springer,2002.
[7] ZHANG Q,LIU Z,GUO B,et al.Geometry-Driven Photorealistic Facial Expression Synthesis[J].IEEE Transactions on Visualization & Computer Graphics,2005,12(1):48-60.
[8] JOSHI P,TIEN W C,DESBRUN M,et al.Learning controls for blendshape based realistic facial animation[C]//Proceedings of ACM SIGGRAPH Eurographics Symposium on Computer Aimation.2003:187-192.
[9] PARK B,CHUNG H,NISHITA T,et al.A feature-based approach to facial expression cloning:Virtual Humans and Social Agents[J].Computer Animation and Virtual Worlds,2005,16(3/4):291-303.
[10] JOSHI P,TIEN W C,DESBRUN M,et al.Learing Cnotrols for Blend Shape Based Realistic Facial Animation[C]//ACM Transactions on Graphics.2006:426-433.
[11] GARRIDO P,VALGAERTS L,REHMSEN O,et al.Automatic face reenactment[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2014:4217-4224.
[12] WESIE T,BOUAZIZ S,LI H,et al.Realtime performance-based facial animation[J].ACM Transactions on Graphics,2011,30(4):1.
[13] CAO C,WENG Y,LIN S,et al.3D shape regression for real-time facial animation[J].ACM Transactions on Graphics,2013,32(4):1.
[14] HUANG X Q,LIN Y X,SONG M L.Three-dimensional facial expression synthesis method based on nonlinear joint learning[J].Journal of Computer-Aided Design & Computer Graphics,2011,23(2).
[15] WILLIAMS L.Performance-driven facial animation [C]//ACM SIGGRAPH Computer Graphics.1990:235-242.
[16] YANG F,WANG J,SHECHTMANE,et al.Expression flow for 3D-aware face component transfer[J].ACM Transactions on Graphics,2011,30(4):1.
[17] PEREZ P,GANGNET M,BLAKE A.Poisson image editing[J].ACM Transactions on Graphics,2003,22(3):313-318.
[18] BITOUK D.Face Swapping:Automatically Replacing Faces in Photographs[J].ACM SIGGRAPH,2008,27(3):1-8.
[19] DALE K,SUNKAVALLI K,JOHNSON M K,et al.Video face replacement[J].ACM Transactions on Graphics,2011,30(6):1.
[20] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative Adversarial Nets[C]//International Conference on Neural Information Processing Systems.MIT Press,2014:2672-2680.
[21] KAZEMI V,SULLIVAN J.One Millisecond Face Alignmentwith an Ensemble of Regression Trees[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2014:1867-1874.
[22] JOHNSON J,ALAHI A,LI F F.Perceptual Losses for Real-Time Style Transfer and Super-Resolution[M]//Computer Vision-ECCV 2016.Springer International Publishing,2016:694-711.
[23] IOFFE S,SZEGEDY C.Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//International Conference on International Conference on Machine Learning.JMLR.org,2015.
[24] ISOPLA P,ZHU J Y,ZHOU T,et al.Image-to-Image Translation with Conditional Adversarial Networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2017.
[25] KINFMA D P,BA J.Adam:A Method for Stochastic Optimization[J].arXiv:1412.6980v8,2014.
[26] HAQ S,JACKSON P J.Multimodal emotion recognition[M]//Machine Audition:Principles,Algorithms and Systems,2010,17:398-423.
[27] CHOI Y,CHOI M,KIM M,et al.StarGAN:Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation[C]//CVPR.2017.
[28] ABADI M,AGARWAL A,BARHAM P,et al.Tensorflow:Large-scale machine learning on heterogeneous distributed systems[J].arXiv:1603.04467,2016.
[1] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[2] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[3] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[4] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[5] HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[6] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[7] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[8] HU Yan-yu, ZHAO Long, DONG Xiang-jun. Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification [J]. Computer Science, 2022, 49(7): 73-78.
[9] CHENG Cheng, JIANG Ai-lian. Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction [J]. Computer Science, 2022, 49(7): 120-126.
[10] HOU Yu-tao, ABULIZI Abudukelimu, ABUDUKELIMU Halidanmu. Advances in Chinese Pre-training Models [J]. Computer Science, 2022, 49(7): 148-163.
[11] ZHOU Hui, SHI Hao-chen, TU Yao-feng, HUANG Sheng-jun. Robust Deep Neural Network Learning Based on Active Sampling [J]. Computer Science, 2022, 49(7): 164-169.
[12] SU Dan-ning, CAO Gui-tao, WANG Yan-nan, WANG Hong, REN He. Survey of Deep Learning for Radar Emitter Identification Based on Small Sample [J]. Computer Science, 2022, 49(7): 226-235.
[13] WANG Jun-feng, LIU Fan, YANG Sai, LYU Tan-yue, CHEN Zhi-yu, XU Feng. Dam Crack Detection Based on Multi-source Transfer Learning [J]. Computer Science, 2022, 49(6A): 319-324.
[14] CHU Yu-chun, GONG Hang, Wang Xue-fang, LIU Pei-shun. Study on Knowledge Distillation of Target Detection Algorithm Based on YOLOv4 [J]. Computer Science, 2022, 49(6A): 337-344.
[15] LIU Wei-ye, LU Hui-min, LI Yu-peng, MA Ning. Survey on Finger Vein Recognition Research [J]. Computer Science, 2022, 49(6A): 1-11.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.