Computer Science ›› 2020, Vol. 47 ›› Issue (9): 142-149.doi: 10.11896/jsjkx.190900203

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Expression Animation Synthesis Based on Improved CycleGan Model and Region Segmentation

YE Ya-nan1,2, CHI Jing1,2, YU Zhi-ping1,2, ZHAN Yu-li1,2and ZHANG Cai-ming1,2,3,4   

  1. 1 School of Computer Science and Technology,Shandong University of Finance and Economics,Jinan 250014,China
    2 Shandong Provincial Key Laboratory of Digital Media Technology,Jinan 250014,China
    3 School of Software,Shandong University,Jinan 250101,China
    4 Future Intelligent Computing Collaborative Innovation Center,Yantai,Shandong 264003,China
  • Received:2019-06-16 Published:2020-09-10
  • About author:YE Ya-nan,born in 1994,master,postgraduate.Her main research interests include computer animation and digital image processing.
    CHI Jing,born in 1980,Ph.D,associate professor,postgraduate supervisor.Her main research interests includecompu-ter animation,geometric shape,and me-dical image processing.
  • Supported by:
    Natural Science Foundation of Shandong Province for Excellent Young Scholars in Provincial Universities (ZR2018JL022),National Natural Science Foundation of China (61772309,61602273),Shandong Provincial Key R&D Program (2019GSF109112),Science and Technology Program of Shandong Education Department (J18RA272) and Fostering Project of Dominant Discipline and Talent Team of Shandong Province Higher Education Institutions.

Abstract: Aiming at the problems of mostly relying on data source driver,low generation efficiency and poor authenticity of the existing facial expression synthesis methods,this paper proposes a new method for expression animation synthesis based on the improved CycleGan model and region segmentation.This new method can synthesize new expression in real time and has good stability and robustness.The proposed method constructs a new covariance constraint in the cycle consistent loss function of the traditional CycleGan model,which can effectively avoid color anomalies and image blurring in generation of new expression images.The idea of zonal training is put forward.The Dlib face recognition database is used to detect the key points of the face images.The detected key feature points are used to segment the face in domain source and target domain into four zones:left eye,right eye,mouth and the rest of the face.The improved CycleGan model is used to train each region separately,and finally the training results are weighted and fused into the final new expression image.The zonal training further enhances the authenticity of expression synthesis.The experimental data comes from the SAVEE database,and the experimental results are presented with python 3.4 software under the Tensorflow framework.Experiments show that the new method can directly generate real and natu-ral new expression sequences in real time on the original facial expression sequence without data source driver.Furthermore,for the voice video,it can effectively ensure the synchronization between the generated facial expression sequence and the source audio.

Key words: Facial expression synthesis, Region segmentation, CycleGan, Covariance constraint, Deep learning

CLC Number: 

  • TP391.41
[1] ZHU J,PARK T,ISOLA P,et al.Unpaired Image-to-ImageTranslation Using Cycle-Consistent Adversarial Networks[C]//2017 IEEE International Conference on Computer Vision (ICCV).2017:2242-2251.
[2] PIGHIN F,HECKER J,LISCHINSKI D,et al.Synthesizing Realistic Facial Expressions from Photographs[C]//Proceedings of the ACM SIGGRAPH Conference on Computer Graphics.1998:75-84.
[3] BLANZ V,BASSO C,VETEER T,et al.Reanimating Faces in Images and Video[C]//European Association for Computer Graphics.2003:641-650.
[4] VLASIC D,BRAND M,PFISTER H,et al.Face Transfer with Multilinear Models[J].ACM Transactions on Graphics,2006,24(3):426-433.
[5] LV P,XU M L.Expression of Face Expre-ssions Unrelated to Expression Database[J].Journal of Computer-Aided Design & Computer Graphics,2016,28(1).
[6] PASQUARIELLO S,PELACHAUD C.GRETA:A Simple Facial Animation Engine[M]//Soft Computing and Industry.London:Springer,2002.
[7] ZHANG Q,LIU Z,GUO B,et al.Geometry-Driven Photorealistic Facial Expression Synthesis[J].IEEE Transactions on Visualization & Computer Graphics,2005,12(1):48-60.
[8] JOSHI P,TIEN W C,DESBRUN M,et al.Learning controls for blendshape based realistic facial animation[C]//Proceedings of ACM SIGGRAPH Eurographics Symposium on Computer Aimation.2003:187-192.
[9] PARK B,CHUNG H,NISHITA T,et al.A feature-based approach to facial expression cloning:Virtual Humans and Social Agents[J].Computer Animation and Virtual Worlds,2005,16(3/4):291-303.
[10] JOSHI P,TIEN W C,DESBRUN M,et al.Learing Cnotrols for Blend Shape Based Realistic Facial Animation[C]//ACM Transactions on Graphics.2006:426-433.
[11] GARRIDO P,VALGAERTS L,REHMSEN O,et al.Automatic face reenactment[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2014:4217-4224.
[12] WESIE T,BOUAZIZ S,LI H,et al.Realtime performance-based facial animation[J].ACM Transactions on Graphics,2011,30(4):1.
[13] CAO C,WENG Y,LIN S,et al.3D shape regression for real-time facial animation[J].ACM Transactions on Graphics,2013,32(4):1.
[14] HUANG X Q,LIN Y X,SONG M L.Three-dimensional facial expression synthesis method based on nonlinear joint learning[J].Journal of Computer-Aided Design & Computer Graphics,2011,23(2).
[15] WILLIAMS L.Performance-driven facial animation [C]//ACM SIGGRAPH Computer Graphics.1990:235-242.
[16] YANG F,WANG J,SHECHTMANE,et al.Expression flow for 3D-aware face component transfer[J].ACM Transactions on Graphics,2011,30(4):1.
[17] PEREZ P,GANGNET M,BLAKE A.Poisson image editing[J].ACM Transactions on Graphics,2003,22(3):313-318.
[18] BITOUK D.Face Swapping:Automatically Replacing Faces in Photographs[J].ACM SIGGRAPH,2008,27(3):1-8.
[19] DALE K,SUNKAVALLI K,JOHNSON M K,et al.Video face replacement[J].ACM Transactions on Graphics,2011,30(6):1.
[20] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative Adversarial Nets[C]//International Conference on Neural Information Processing Systems.MIT Press,2014:2672-2680.
[21] KAZEMI V,SULLIVAN J.One Millisecond Face Alignmentwith an Ensemble of Regression Trees[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2014:1867-1874.
[22] JOHNSON J,ALAHI A,LI F F.Perceptual Losses for Real-Time Style Transfer and Super-Resolution[M]//Computer Vision-ECCV 2016.Springer International Publishing,2016:694-711.
[23] IOFFE S,SZEGEDY C.Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//International Conference on International Conference on Machine,2015.
[24] ISOPLA P,ZHU J Y,ZHOU T,et al.Image-to-Image Translation with Conditional Adversarial Networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2017.
[25] KINFMA D P,BA J.Adam:A Method for Stochastic Optimization[J].arXiv:1412.6980v8,2014.
[26] HAQ S,JACKSON P J.Multimodal emotion recognition[M]//Machine Audition:Principles,Algorithms and Systems,2010,17:398-423.
[27] CHOI Y,CHOI M,KIM M,et al.StarGAN:Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation[C]//CVPR.2017.
[28] ABADI M,AGARWAL A,BARHAM P,et al.Tensorflow:Large-scale machine learning on heterogeneous distributed systems[J].arXiv:1603.04467,2016.
[1] WANG Rui-ping, JIA Zhen, LIU Chang, CHEN Ze-wei, LI Tian-rui. Deep Interest Factorization Machine Network Based on DeepFM [J]. Computer Science, 2021, 48(1): 226-232.
[2] YU Wen-jia, DING Shi-fei. Conditional Generative Adversarial Network Based on Self-attention Mechanism [J]. Computer Science, 2021, 48(1): 241-246.
[3] TONG Xin, WANG Bin-jun, WANG Run-zheng, PAN Xiao-qin. Survey on Adversarial Sample of Deep Learning Towards Natural Language Processing [J]. Computer Science, 2021, 48(1): 258-267.
[4] DING Yu, WEI Hao, PAN Zhi-song, LIU Xin. Survey of Network Representation Learning [J]. Computer Science, 2020, 47(9): 52-59.
[5] HE Xin, XU Juan, JIN Ying-ying. Action-related Network:Towards Modeling Complete Changeable Action [J]. Computer Science, 2020, 47(9): 123-128.
[6] DENG Liang, XU Geng-lin, LI Meng-jie, CHEN Zhang-jin. Fast Face Recognition Based on Deep Learning and Multiple Hash Similarity Weighting [J]. Computer Science, 2020, 47(9): 163-168.
[7] BAO Yu-xuan, LU Tian-liang, DU Yan-hui. Overview of Deepfake Video Detection Technology [J]. Computer Science, 2020, 47(9): 283-292.
[8] YUAN Ye, HE Xiao-ge, ZHU Ding-kun, WANG Fu-lee, XIE Hao-ran, WANG Jun, WEI Ming-qiang, GUO Yan-wen. Survey of Visual Image Saliency Detection [J]. Computer Science, 2020, 47(7): 84-91.
[9] WANG Wen-dao, WANG Run-ze, WEI Xin-lei, QI Yun-liang, MA Yi-de. Automatic Recognition of ECG Based on Stacked Bidirectional LSTM [J]. Computer Science, 2020, 47(7): 118-124.
[10] LIU Yan, WEN Jing. Complex Scene Text Detection Based on Attention Mechanism [J]. Computer Science, 2020, 47(7): 135-140.
[11] ZHANG Zhi-yang, ZHANG Feng-li, TAN Qi, WANG Rui-jin. Review of Information Cascade Prediction Methods Based on Deep Learning [J]. Computer Science, 2020, 47(7): 141-153.
[12] JIANG Wen-bin, FU Zhi, PENG Jing, ZHU Jian. 4Bit-based Gradient Compression Method for Distributed Deep Learning System [J]. Computer Science, 2020, 47(7): 220-226.
[13] CHEN Jin-yin, ZHANG Dun-Jie, LIN Xiang, XU Xiao-dong and ZHU Zi-ling. False Message Propagation Suppression Based on Influence Maximization [J]. Computer Science, 2020, 47(6A): 17-23.
[14] CHENG Zhe, BAI Qian, ZHANG Hao, WANG Shi-pu and LIANG Yu. Improving Hi-C Data Resolution with Deep Convolutional Neural Networks [J]. Computer Science, 2020, 47(6A): 70-74.
[15] HE Lei, SHAO Zhan-peng, ZHANG Jian-hua and ZHOU Xiao-long. Review of Deep Learning-based Action Recognition Algorithms [J]. Computer Science, 2020, 47(6A): 139-147.
Full text



[1] LEI Li-hui and WANG Jing. Parallelization of LTL Model Checking Based on Possibility Measure[J]. Computer Science, 2018, 45(4): 71 -75 .
[2] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[3] ZHANG Jia-nan and XIAO Ming-yu. Approximation Algorithm for Weighted Mixed Domination Problem[J]. Computer Science, 2018, 45(4): 83 -88 .
[4] WU Jian-hui, HUANG Zhong-xiang, LI Wu, WU Jian-hui, PENG Xin and ZHANG Sheng. Robustness Optimization of Sequence Decision in Urban Road Construction[J]. Computer Science, 2018, 45(4): 89 -93 .
[5] ZHOU Yan-ping and YE Qiao-lin. L1-norm Distance Based Least Squares Twin Support Vector Machine[J]. Computer Science, 2018, 45(4): 100 -105 .
[6] LIU Bo-yi, TANG Xiang-yan and CHENG Jie-ren. Recognition Method for Corn Borer Based on Templates Matching in Muliple Growth Periods[J]. Computer Science, 2018, 45(4): 106 -111 .
[7] GENG Hai-jun, SHI Xin-gang, WANG Zhi-liang, YIN Xia and YIN Shao-ping. Energy-efficient Intra-domain Routing Algorithm Based on Directed Acyclic Graph[J]. Computer Science, 2018, 45(4): 112 -116 .
[8] WANG Zhen-chao, HOU Huan-huan and LIAN Rui. Path Optimization Scheme for Restraining Degree of Disorder in CMT[J]. Computer Science, 2018, 45(4): 122 -125 .
[9] YANG Yu-qi, ZHANG Guo-an and JIN Xi-long. Dual-cluster-head Routing Protocol Based on Vehicle Density in VANETs[J]. Computer Science, 2018, 45(4): 126 -130 .
[10] SHI Chao, XIE Zai-peng, LIU Han and LV Xin. Optimization of Container Deployment Strategy Based on Stable Matching[J]. Computer Science, 2018, 45(4): 131 -136 .