计算机科学 ›› 2019, Vol. 46 ›› Issue (9): 22-27.doi: 10.11896/j.issn.1002-137X.2019.09.003

• 综述 • 上一篇    下一篇

基于神经网络的角色运动合成研究进展

王鑫1,2, 孟浩浩1,2, 姜小涛1,2, 陈胜勇1,3, 孙凌云4,5   

  1. (浙江工业大学计算机科学与技术学院 杭州310023)1;
    (浙江省可视媒体智能处理技术研究重点实验室 杭州310023)2;
    (天津理工大学计算机科学与工程学院 天津300384)3;
    (浙江大学现代工业设计研究院 杭州310058)4;
    (浙江大学国际设计研究院 杭州310058)5
  • 收稿日期:2018-11-14 出版日期:2019-09-15 发布日期:2019-09-02
  • 通讯作者: 王 鑫(1984-),男,博士,副教授,CCF会员,主要研究方向为计算机视觉、计算机图形学,E-mail:xinw@zjut.edu.cn
  • 作者简介:孟浩浩(1993-),男,硕士生,主要研究方向为计算机图形学;姜小涛(1992-),男,硕士生,主要研究方向为计算机图形学;陈胜勇(1973-),男,博士,教授,博士生导师,主要研究方向为计算机视觉、图像分析与处理、机器人智能技术;孙凌云(1981-),男,博士,教授,博士生导师,主要研究方向为人工智能、设计智能、信息与交互设计。
  • 基金资助:
    国家自然科学基金(61303142,61004116,61672451,U1509207),浙江省自然科学基金(Y1110882)

Survey on Character Motion Synthesis Based on Neural Network

WANG Xin1,2, MENG Hao-hao1,2, JIANG Xiao-tao1,2, CHEN Sheng-yong1,3, SUN Ling-yun4,5   

  1. (College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China)1;
    (Key Laboratory of Visual Media Intelligent Process Technology of Zhejiang Province,Hangzhou 310023,China)2;
    (College of Computer Science and Engineering,Tianjin University of Technology,Tianjin 300384,China)3;
    (Modern Industrial Design Institute,Zhejiang University,Hangzhou 310058,China)4;
    (International Design Institute,Zhejiang University,Hangzhou 310058,China)5
  • Received:2018-11-14 Online:2019-09-15 Published:2019-09-02

摘要: 在角色运动数据集上,运用神经网络技术进行运动合成是当前计算机图形学领域中的一项重要研究。该研究旨在通过神经网络技术生成自然、逼真度较高的角色运动。在对相关研究工作进行分析和总结的基础上,对运动模型的构建、运动交互和运动风格化等领域的研究进展进行了介绍;详细阐述了基于运动捕获数据,利用数据驱动技术、交互式控制方法和ERD,CAE,MAR等网络模型,动态地对角色进行运动建模、运动合成、交互式运动控制,同时为了合成更高质量的角色运动,对运动动画进行风格化等处理;以神经网络技术为着眼点,串联角色运动合成中的各个环节,并结合实际应用,针对当前研究工作面临的难点提出一些可继续深入探索的问题。

关键词: 角色运动合成, 神经网络, 数据驱动, 交互式角色控制, 风格编辑

Abstract: The application of neural network technology to character motion synthesis on human motion data sets is an important research content in the field of computer graphics.This study aims to generate naturally realistic character motion using neural networks through date-driven technology.Based on the analysis and summary of related research work,this paper introduced the research progress in the fields of motion model construction,motion interaction and motion stylization and so on.Based on the motion capture data,by using data-driven technology,interactive control methods and network models such as ERD,CAE and MAR,the character was dynamically modeled,synthesized and controlled by interactive motion,and in order to generate higher quality character motions,motion animation and other content were stylized.In this paper,taking neural network technology as the focal point,various study works of the character motion synthesis were connected.Combined with the practical applications and difficulties faced in the current research work,this paper suggested some problems that can be further studied.

Key words: Character motion synthesis, Neural network, Data driven, Interactive character control, Style editting

中图分类号: 

  • TP311
[1]GUO S,SOUTHERN R,CHANG J,et al.Adaptive motion synthesis for virtual characters:a survey[J].The Visual Computer,2015,31(5):497-512.
[2]WANG X,CHEN Q,WANG W.3D Human Motion Editing and Synthesis:A Survey[J].Computational and Mathematical Methods in Medicine,2014,2014(3):104535.
[3]LIU G D,PANG Z G,CHENG X,et al.A Survey on Machine Learning in the Synthesis of Human Motions[J].Journal of Computer-Aided Design and Computer Graphics,2010,22(9):1619-1627.(in Chinese)刘更代,潘志庚,程熙,等.人体运动合成中的机器学习技术综述[J].计算机辅助设计与图形学学报,2010,22(9):1619-1627.
[4]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[J].Advances in Neural Information Processing Systems,2012,25(2):1097-1105.
[5]SHAZEER N,MIRHOSEINI A,MAZIARZ K,et al.Outra-geously large neural networks:The sparsely-gated mixture-of-experts layer[J].arXiv:1701.06538,2017.
[6]PENG X B,ABBEEL P,LEVINE S,et al.DeepMimic:Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills[J].arXiv:1804.02717,2018.
[7]GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//International Conference on Neural Information Processing Systems.MIT Press,Montreal,Canada,2014:2672-2680.
[8]GENG J,SHAO T,ZHENG Y,et al.Warp-guided GANs for single-photo facial animation[C]//SIGGRAPH Asia 2018 Technical Papers.ACM,Tokyo,Japan,2018:231.
[9]KELLY T,GUERRERO P,STEED A,et al.FrankenGAN:Guided Detail Synthesis for Building Mass-Models Using Style-Synchonized GANs[J].arXiv:1806.07179,2018.
[10]BENGIO E,BACON P L,PINEAU J,et al.Conditional computation in neural networks for faster models[J].arXiv:1511.06297,2015.
[11]FU J,CO-REYES J,LEVINE S.Ex2:Exploration with exem-plar models for deep reinforcement learning[C]//Advances in Neural Information Processing Systems.Long Beach,USA,2017:2577-2587.
[12]ZHAO J J,WEI Y,XIA S H,et al.Survey of Physics-BasedCharacter Animation [J].Journal of Computer Research and Development,2015,52(12):2866-2878.(in Chinese)赵建军,魏毅,夏时洪,等.基于物理的角色动画合成方法综述[J].计算机研究与发展,2015,52(12):2866-2878.
[13]ALLEN B F,FALOUTSOS P.Evolved Controllers for Simulated Locomotion[C]//International Workshop on Motion in Games.Springer-Verlag,2009:219-230.
[14]TAN J,GU Y,LIU C K,et al.Learning bicycle stunts[J].Acm Transactions on Graphics,2014,33(4):50.
[15]MORDATCH I,LOWREY K,ANDREW G,et al.Interactive control of diverse complex characters with neural networks[C]//Advances in Neural Information Processing Systems.Long Beach,USA,2015:3132-3140.
[16]LEVINE S,KOLTUN V.Learning Complex Neural Network Policies with Trajectory Optimization[C]//International Conference on Machine Learning.Beijing,China,2014:829-837.
[17]DU Y,WANG W,WANG L.Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston,USA,2015:1110-1118.
[18]HOLDEN D,SAITO J,KOMURA T,et al.Learning motionmanifolds with convolutional autoencoders[C]//SIGGRAPH Asia 2015 Technical Briefs.Kobe,Japan,2015:18.
[19]HOLDEN D,KOMURA T,SAITO J.Phase-functioned neural networks for character control[J].Acm Transactions on Gra-phics,2017,36(4):1-13.
[20]HOLDEN D,SAITO J,KOMURA T.A deep learning framework for character motion synthesis and editing[J].Acm Transactions on Graphics,2016,35(4):1-11.
[21]XIA S H,WEI Y,WANG Z Q.A Survey of Physics-Based Human Motion Simulation[J].Journal of Computer Research and Development,2010,47(8):1354-1361.(in Chinese)夏时洪,魏毅,王兆其.人体运动仿真综述[J].计算机研究与发展,2010,47(8):1354-1361.
[22]MITTELMAN R,KUIPERS B,SAVARESE S,et al.Structuredrecurrent temporal restricted boltzmann machines[C]//International Conference on Machine Learning.JMLR.org,2014:II-1647.
[23]FRAGKIADAKI K,LEVINE S,FELSEN P,et al.Recurrentnetwork models for human dynamics[C]//Proceedings of the IEEE International Conference on Computer Vision.San Diego,USA,2015:4346-4354.
[24]SAITO S,HU L,MA C,et al.3D hair synthesis using volumetric variational autoencoders[C]//SIGGRAPH Asia 2018 Technical Papers.Tokyo,Japan,2018:208.
[25]CHU M,THUEREY N.Data-driven synthesis of smoke flows with CNN-based feature descriptors[J].ACM Transactions on Graphics (TOG),2017,36(4):69.
[26]XIA S,WANG C,CHAI J,et al.Realtime style transfer for unlabeled heterogeneous human motion[J].ACM Transactions on Graphics,2015,34(4):1-10.
[27]LI Z,ZHOU Y,XIAO S,et al.Auto-conditioned recurrent networks for extended complex human motion synthesis[J].arXiv:1707.05363,2017.
[28]BARSOUM E,KENDER J,LIU Z.HP-GAN:Probabilistic 3Dhuman motion prediction via GAN[J].arXiv:1711.09561,2017.
[29]RAJESWARAN A,KUMAR V,GUPTA A,et al.Learningcomplex dexterous manipulation with deep reinforcementlear-ning and demonstrations[J].arXiv:1709.10087,2017.
[30]MA P,TIAN Y,PAN Z,et al.Fluid directed rigid body control using deep reinforcement learning[J].ACM Transactions on Graphics (TOG),2018,37(4):96.
[31]HUANG T C,HUANG Y J,LIN W C.Real-time horse gaitsynthesis[J].Computer Animation and Virtual Worlds,2013,24(2):87-95.
[32]ZHANG H,STARKE S,KOMURA T,et al.Mode-adaptiveneural networks for quadruped motion control[J].ACM Tran-sactions on Graphics (TOG),2018,37(4):145.
[33]LEE K,LEE S,LEE J.Interactive character animation by lear-ning multi-objective control[C]//SIGGRAPH Asia 2018 Technical Papers.Tokyo,Japan,2018:180.
[34]HO J,ERMON S.Generative adversarial imitation learning[C]//Advances in Neural Information Processing Systems.Barcelona,Spain,2016:4565-4573.
[35]LIU L,HODGINS J.Learning to schedule control fragments for physics-based characters using deep q-learning[J].ACM Transactions on Graphics (TOG),2017,36(3):29.
[36]LAU M,KUFFNER J J.Behavior planning for character animation[C]//ACM Siggraph/eurographics Symposium on ComputerAnimation.Los Angeles,USA,2005:271-280.
[37]SAFONOVA A,HODGINS J K.Construction and optimalsearch of interpolated motion graphs[J].ACM Transactions on Graphics,2007,26(3):106.
[38]LIU L,YIN K K,VAN DE PANNE M,et al.Sampling-based contact-rich motion control[C]//ACM Transactions on Gra-phics (TOG).New York,USA,2010,29(4):128.
[39]MIN J,CHAI J.Motion graphs++:a compact generative model for semantic motion analysis and synthesis[J].ACM Transactions on Graphics (TOG),2012,31(6):153.
[40]HWANG J,KIM J,SUH I H,et al.Real-time Locomotion Controller using an Inverted-Pendulum-based Abstract Model[J].Computer Graphics Forum,2018,37(2):287-296.
[41]RAJESWARAN A,GHOTRA S,RAVINDRAN B,et al.Epopt:Learning robust neural network policies using model ensembles[J].arXiv:1610.01283,2016.
[42]SU W,DU D,YANG X,et al.Interactive sketch-based normal map generation with deep neural networks[J].Proceedings of the ACM on Computer Graphics and Interactive Techniques,2018,1(1):22.
[43]LEVINE S,WANG J M,HARAUX A,et al.Continuous character control with low-dimensional embeddings[J].ACM Tran-sactions on Graphics (TOG),2012,31(4):28.
[44]CLEGG A,YU W,TAN J,et al.Learning to dress:synthesizing human dressing motion via deep reinforcement learning[C]//SIGGRAPH Asia 2018 Technical Papers.Tokyo,Japan,2018:179.
[45]PENG X B,BERSETH G,VAN DE PANNE M.Terrain-adaptive locomotion skills using deep reinforcement learning[J].ACM Transactions on Graphics (TOG),2016,35(4):81.
[46]LIU L,HODGINS J.Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning[J].ACM Transactions on Graphics (TOG),2018,37(4):142.
[47]NAKADA M,CHEN H,TERZOPOULOS D.Learning Biomi-metic Perception for Human Sensorimotor Control[C]//Procee-dings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Salt Lake City,USA,2018:1917-1922.
[48]NAKADA M,ZHOU T,CHEN H,et al.Deep learning of biomimetic sensorimotor control for biomechanical human animation[J].ACM Transactions on Graphics (TOG),2018,37(4):56.
[49]NAIR A,MCGREW B,ANDRYCHOWICZ M,et al.Overco-ming exploration in reinforcement learning with demonstrations[C]//2018 IEEE International Conference on Robotics and Automation (ICRA).IEEE,Brisbane,Australis,2018:6292-6299.
[50]GUPTA A,SATKIN S,EFROS A A,et al.From 3d scene geometry to human workspace[C]//2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2011:1961-1968.
[51]KIM V G,CHAUDHURI S,GUIBAS L,et al.Shape2pose:Human-centric shape analysis[J].ACM Transactions on Graphics (TOG),2014,33(4):120.
[52]SAVVA M,CHANG A X,Hanrahan P,et al.PiGraphs:learning interaction snapshots from observations[J].ACM Transactions on Graphics (TOG),2016,35(4):139.
[53]KAPADIA M,XU X H,NITTI M,et al.Precision:Precompu-ting environment semantics for contact-rich character animation[C]//Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games.ACM,2016:29-37.
[54]CHANG X,HOSPEDALES T M,XIANG T.Multi-level factorisation net for person re-identification[C]//CVPR.Salt Lake City,USA,2018:2.
[55]KULKARNI T D,WHITNEY W F,KOHLI P,et al.Deep convolutional inverse graphics network[C]//Advances in Neural Information Processing Systems.Long Beach,USA,2015:2539-2547.
[56]CHOI B,LEWIS J P,SEOL Y,et al.SketchiMo:sketch-basedmotion editing for articulated characters[J].ACM Transactions on Graphics (TOG),2016,35(4):146.
[57]NEFF M,FIUME E.Aesthetic edits for character animation[C]//Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation.San Diego,California:ACM,2003.
[58]COLEMAN P,BIBLIOWICZ J,SINGH K,et al.Staggered poses:a character motion representation for detail-preserving editing of pose and coordinated timing[C]//Eurographics/ACM SIGGRAPH Symposium on Computer Animation,SCA 2008.Dublin,Ireland,DBLP,2008:137-146.
[59]GUAY M,GLEICHER M,CANI M P.Adding dynamics tosketch-based character animations[C]//Sketch-Based Interfaces and Modeling.2015:27-34.
[60]IKEMOTO L,ARIKAN O,FORSYTH D.Generalizing motion edits with Gaussian processes[J].Acm Transactions on Graphi-cs,2009,28(1):1-12.
[61]HSU E,PULLI K.Style translation for human motion[J].ACM Transactions on Graphics (TOG),2005,24(3):1082-1089.
[62]GATYS L A,ECKER S,BETHGE M.A neural algorithm of artistic style[J].arXiv:1508.06576,2015.
[63]GATYS L A,ECKER A S,BETHGE M.Image style transfer using convolutional neural networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Las Vegas,USA,2016:2414-2423.
[64]BRAND M,HERTZMANN A.Style machines[C]//Procee-dings of the 27th annual conference on Computer graphics and interactive techniques.ACM Press/Addison-Wesley Publishing Co.,2000:183-192.
[65]MIN J,LIU H,CHAI J.Synthesis and editing of personalizedstylistic human motion[C]//Symposium on Interactive 3d Graphics(Si3d 2010).Washington,DC,USA,DBLP,2010:39-46.
[66]HOLDEN D,HABIBIE I,KUSAJIMA I,et al.Fast NeuralStyle Transfer for Motion Data[J].IEEE computer graphics and applications,2017,37(4):42-49.
[1] 李青华, 李翠平, 张静, 陈红, 王绍卿. 深度神经网络压缩综述[J]. 计算机科学, 2019, 46(9): 1-14.
[2] 王嫣然, 陈清亮, 吴俊君. 面向复杂环境的图像语义分割方法综述[J]. 计算机科学, 2019, 46(9): 36-46.
[3] 孙中锋, 王静. 用于基于方面情感分析的RCNN-BGRU-HN网络模型[J]. 计算机科学, 2019, 46(9): 223-228.
[4] 缪永伟, 李高怡, 鲍陈, 张旭东, 彭思龙. 基于卷积神经网络的图像局部风格迁移[J]. 计算机科学, 2019, 46(9): 259-264.
[5] 江泽涛, 秦嘉奇, 胡硕. 基于多路卷积神经网络的多光谱场景识别方法[J]. 计算机科学, 2019, 46(9): 265-270.
[6] 石晓红, 黄钦开, 苗佳欣, 苏卓. 基于卷积网络的边缘保持滤波方法[J]. 计算机科学, 2019, 46(9): 277-283.
[7] 周燕, 曾凡智, 吴臣, 罗粤, 刘紫琴. 基于深度学习的三维形状特征提取方法[J]. 计算机科学, 2019, 46(9): 47-58.
[8] 郭旭, 朱敬华. 基于用户向量化表示和注意力机制的深度神经网络推荐模型[J]. 计算机科学, 2019, 46(8): 111-115.
[9] 于洋, 李世杰, 陈亮, 刘韵婷. 基于改进YOLO v2的船舶目标检测方法[J]. 计算机科学, 2019, 46(8): 332-336.
[10] 刘子奇, 刘世光. 风格线条画生成技术综述[J]. 计算机科学, 2019, 46(7): 13-21.
[11] 麦应潮,陈云华,张灵. 具有生物真实性的强抗噪性神经元激活函数[J]. 计算机科学, 2019, 46(7): 206-210.
[12] 王乐乐,汪斌强,刘建港,张建辉,苗启广. 基于递归神经网络的恶意程序检测研究[J]. 计算机科学, 2019, 46(7): 86-90.
[13] 王雅慧, 刘博, 袁晓彤. 基于近似牛顿法的分布式卷积神经网络训练[J]. 计算机科学, 2019, 46(7): 180-185.
[14] 张琳娜,陈建强,陈晓玲,岑翼刚,阚世超. 面向行车视频目标实时检测的轻量级SSD网络[J]. 计算机科学, 2019, 46(7): 233-237.
[15] 孔繁钰, 周愉峰, 陈纲. 基于时空特征挖掘的交通流量预测方法[J]. 计算机科学, 2019, 46(7): 322-326.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] 厉柏伸,李领治,孙涌,朱艳琴. 基于伪梯度提升决策树的内网防御算法[J]. 计算机科学, 2018, 45(4): 157 -162 .
[2] 王振武,吕小华,韩晓辉. 基于四叉树分割的地形LOD技术综述[J]. 计算机科学, 2018, 45(4): 34 -45 .
[3] 李键红,吴亚榕,吕巨建. 基于组稀疏表示的在线单帧图像超分辨率算法[J]. 计算机科学, 2018, 45(4): 312 -318 .
[4] 周枫, 李荣雨. 基于BGRU池的卷积神经网络文本分类模型[J]. 计算机科学, 2018, 45(6): 235 -240 .
[5] 田晓燕,魏娜,范泽铭,张锁良. RA码交织器的研究与设计[J]. 计算机科学, 2018, 45(5): 79 -82, 88 .
[6] 李俊,罗阳坤,李波,李乔木. 基于异维变异的差分混合粒子群算法[J]. 计算机科学, 2018, 45(5): 208 -214 .
[7] 刘壮,柴秀娟,陈熙霖. 双通道Faster R-CNN在RGB-D手部检测中的应用[J]. 计算机科学, 2018, 45(5): 232 -237 .
[8] 阳小兰,钱程,朱福喜. 基于云计算的大数据服务资源评价方法[J]. 计算机科学, 2018, 45(5): 295 -299 .
[9] 赵星宇, 丁世飞. 深度强化学习研究综述[J]. 计算机科学, 2018, 45(7): 1 -6 .
[10] 何思源,欧博,廖鑫. 分布式工作流环境下角色匹配的访问控制模型[J]. 计算机科学, 2018, 45(7): 129 -134 .