计算机科学 ›› 2019, Vol. 46 ›› Issue (9): 22-27.doi: 10.11896/j.issn.1002-137X.2019.09.003

• 综述 • 上一篇    下一篇

基于神经网络的角色运动合成研究进展

王鑫1,2, 孟浩浩1,2, 姜小涛1,2, 陈胜勇1,3, 孙凌云4,5   

  1. (浙江工业大学计算机科学与技术学院 杭州310023)1;
    (浙江省可视媒体智能处理技术研究重点实验室 杭州310023)2;
    (天津理工大学计算机科学与工程学院 天津300384)3;
    (浙江大学现代工业设计研究院 杭州310058)4;
    (浙江大学国际设计研究院 杭州310058)5
  • 收稿日期:2018-11-14 出版日期:2019-09-15 发布日期:2019-09-02
  • 通讯作者: 王 鑫(1984-),男,博士,副教授,CCF会员,主要研究方向为计算机视觉、计算机图形学,E-mail:xinw@zjut.edu.cn
  • 作者简介:孟浩浩(1993-),男,硕士生,主要研究方向为计算机图形学;姜小涛(1992-),男,硕士生,主要研究方向为计算机图形学;陈胜勇(1973-),男,博士,教授,博士生导师,主要研究方向为计算机视觉、图像分析与处理、机器人智能技术;孙凌云(1981-),男,博士,教授,博士生导师,主要研究方向为人工智能、设计智能、信息与交互设计。
  • 基金资助:
    国家自然科学基金(61303142,61004116,61672451,U1509207),浙江省自然科学基金(Y1110882)

Survey on Character Motion Synthesis Based on Neural Network

WANG Xin1,2, MENG Hao-hao1,2, JIANG Xiao-tao1,2, CHEN Sheng-yong1,3, SUN Ling-yun4,5   

  1. (College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China)1;
    (Key Laboratory of Visual Media Intelligent Process Technology of Zhejiang Province,Hangzhou 310023,China)2;
    (College of Computer Science and Engineering,Tianjin University of Technology,Tianjin 300384,China)3;
    (Modern Industrial Design Institute,Zhejiang University,Hangzhou 310058,China)4;
    (International Design Institute,Zhejiang University,Hangzhou 310058,China)5
  • Received:2018-11-14 Online:2019-09-15 Published:2019-09-02

摘要: 在角色运动数据集上,运用神经网络技术进行运动合成是当前计算机图形学领域中的一项重要研究。该研究旨在通过神经网络技术生成自然、逼真度较高的角色运动。在对相关研究工作进行分析和总结的基础上,对运动模型的构建、运动交互和运动风格化等领域的研究进展进行了介绍;详细阐述了基于运动捕获数据,利用数据驱动技术、交互式控制方法和ERD,CAE,MAR等网络模型,动态地对角色进行运动建模、运动合成、交互式运动控制,同时为了合成更高质量的角色运动,对运动动画进行风格化等处理;以神经网络技术为着眼点,串联角色运动合成中的各个环节,并结合实际应用,针对当前研究工作面临的难点提出一些可继续深入探索的问题。

关键词: 角色运动合成, 神经网络, 数据驱动, 交互式角色控制, 风格编辑

Abstract: The application of neural network technology to character motion synthesis on human motion data sets is an important research content in the field of computer graphics.This study aims to generate naturally realistic character motion using neural networks through date-driven technology.Based on the analysis and summary of related research work,this paper introduced the research progress in the fields of motion model construction,motion interaction and motion stylization and so on.Based on the motion capture data,by using data-driven technology,interactive control methods and network models such as ERD,CAE and MAR,the character was dynamically modeled,synthesized and controlled by interactive motion,and in order to generate higher quality character motions,motion animation and other content were stylized.In this paper,taking neural network technology as the focal point,various study works of the character motion synthesis were connected.Combined with the practical applications and difficulties faced in the current research work,this paper suggested some problems that can be further studied.

Key words: Character motion synthesis, Neural network, Data driven, Interactive character control, Style editting

中图分类号: 

  • TP311
[1]GUO S,SOUTHERN R,CHANG J,et al.Adaptive motion synthesis for virtual characters:a survey[J].The Visual Computer,2015,31(5):497-512.
[2]WANG X,CHEN Q,WANG W.3D Human Motion Editing and Synthesis:A Survey[J].Computational and Mathematical Methods in Medicine,2014,2014(3):104535.
[3]LIU G D,PANG Z G,CHENG X,et al.A Survey on Machine Learning in the Synthesis of Human Motions[J].Journal of Computer-Aided Design and Computer Graphics,2010,22(9):1619-1627.(in Chinese)刘更代,潘志庚,程熙,等.人体运动合成中的机器学习技术综述[J].计算机辅助设计与图形学学报,2010,22(9):1619-1627.
[4]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[J].Advances in Neural Information Processing Systems,2012,25(2):1097-1105.
[5]SHAZEER N,MIRHOSEINI A,MAZIARZ K,et al.Outra-geously large neural networks:The sparsely-gated mixture-of-experts layer[J].arXiv:1701.06538,2017.
[6]PENG X B,ABBEEL P,LEVINE S,et al.DeepMimic:Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills[J].arXiv:1804.02717,2018.
[7]GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//International Conference on Neural Information Processing Systems.MIT Press,Montreal,Canada,2014:2672-2680.
[8]GENG J,SHAO T,ZHENG Y,et al.Warp-guided GANs for single-photo facial animation[C]//SIGGRAPH Asia 2018 Technical Papers.ACM,Tokyo,Japan,2018:231.
[9]KELLY T,GUERRERO P,STEED A,et al.FrankenGAN:Guided Detail Synthesis for Building Mass-Models Using Style-Synchonized GANs[J].arXiv:1806.07179,2018.
[10]BENGIO E,BACON P L,PINEAU J,et al.Conditional computation in neural networks for faster models[J].arXiv:1511.06297,2015.
[11]FU J,CO-REYES J,LEVINE S.Ex2:Exploration with exem-plar models for deep reinforcement learning[C]//Advances in Neural Information Processing Systems.Long Beach,USA,2017:2577-2587.
[12]ZHAO J J,WEI Y,XIA S H,et al.Survey of Physics-BasedCharacter Animation [J].Journal of Computer Research and Development,2015,52(12):2866-2878.(in Chinese)赵建军,魏毅,夏时洪,等.基于物理的角色动画合成方法综述[J].计算机研究与发展,2015,52(12):2866-2878.
[13]ALLEN B F,FALOUTSOS P.Evolved Controllers for Simulated Locomotion[C]//International Workshop on Motion in Games.Springer-Verlag,2009:219-230.
[14]TAN J,GU Y,LIU C K,et al.Learning bicycle stunts[J].Acm Transactions on Graphics,2014,33(4):50.
[15]MORDATCH I,LOWREY K,ANDREW G,et al.Interactive control of diverse complex characters with neural networks[C]//Advances in Neural Information Processing Systems.Long Beach,USA,2015:3132-3140.
[16]LEVINE S,KOLTUN V.Learning Complex Neural Network Policies with Trajectory Optimization[C]//International Conference on Machine Learning.Beijing,China,2014:829-837.
[17]DU Y,WANG W,WANG L.Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston,USA,2015:1110-1118.
[18]HOLDEN D,SAITO J,KOMURA T,et al.Learning motionmanifolds with convolutional autoencoders[C]//SIGGRAPH Asia 2015 Technical Briefs.Kobe,Japan,2015:18.
[19]HOLDEN D,KOMURA T,SAITO J.Phase-functioned neural networks for character control[J].Acm Transactions on Gra-phics,2017,36(4):1-13.
[20]HOLDEN D,SAITO J,KOMURA T.A deep learning framework for character motion synthesis and editing[J].Acm Transactions on Graphics,2016,35(4):1-11.
[21]XIA S H,WEI Y,WANG Z Q.A Survey of Physics-Based Human Motion Simulation[J].Journal of Computer Research and Development,2010,47(8):1354-1361.(in Chinese)夏时洪,魏毅,王兆其.人体运动仿真综述[J].计算机研究与发展,2010,47(8):1354-1361.
[22]MITTELMAN R,KUIPERS B,SAVARESE S,et al.Structuredrecurrent temporal restricted boltzmann machines[C]//International Conference on Machine Learning.JMLR.org,2014:II-1647.
[23]FRAGKIADAKI K,LEVINE S,FELSEN P,et al.Recurrentnetwork models for human dynamics[C]//Proceedings of the IEEE International Conference on Computer Vision.San Diego,USA,2015:4346-4354.
[24]SAITO S,HU L,MA C,et al.3D hair synthesis using volumetric variational autoencoders[C]//SIGGRAPH Asia 2018 Technical Papers.Tokyo,Japan,2018:208.
[25]CHU M,THUEREY N.Data-driven synthesis of smoke flows with CNN-based feature descriptors[J].ACM Transactions on Graphics (TOG),2017,36(4):69.
[26]XIA S,WANG C,CHAI J,et al.Realtime style transfer for unlabeled heterogeneous human motion[J].ACM Transactions on Graphics,2015,34(4):1-10.
[27]LI Z,ZHOU Y,XIAO S,et al.Auto-conditioned recurrent networks for extended complex human motion synthesis[J].arXiv:1707.05363,2017.
[28]BARSOUM E,KENDER J,LIU Z.HP-GAN:Probabilistic 3Dhuman motion prediction via GAN[J].arXiv:1711.09561,2017.
[29]RAJESWARAN A,KUMAR V,GUPTA A,et al.Learningcomplex dexterous manipulation with deep reinforcementlear-ning and demonstrations[J].arXiv:1709.10087,2017.
[30]MA P,TIAN Y,PAN Z,et al.Fluid directed rigid body control using deep reinforcement learning[J].ACM Transactions on Graphics (TOG),2018,37(4):96.
[31]HUANG T C,HUANG Y J,LIN W C.Real-time horse gaitsynthesis[J].Computer Animation and Virtual Worlds,2013,24(2):87-95.
[32]ZHANG H,STARKE S,KOMURA T,et al.Mode-adaptiveneural networks for quadruped motion control[J].ACM Tran-sactions on Graphics (TOG),2018,37(4):145.
[33]LEE K,LEE S,LEE J.Interactive character animation by lear-ning multi-objective control[C]//SIGGRAPH Asia 2018 Technical Papers.Tokyo,Japan,2018:180.
[34]HO J,ERMON S.Generative adversarial imitation learning[C]//Advances in Neural Information Processing Systems.Barcelona,Spain,2016:4565-4573.
[35]LIU L,HODGINS J.Learning to schedule control fragments for physics-based characters using deep q-learning[J].ACM Transactions on Graphics (TOG),2017,36(3):29.
[36]LAU M,KUFFNER J J.Behavior planning for character animation[C]//ACM Siggraph/eurographics Symposium on ComputerAnimation.Los Angeles,USA,2005:271-280.
[37]SAFONOVA A,HODGINS J K.Construction and optimalsearch of interpolated motion graphs[J].ACM Transactions on Graphics,2007,26(3):106.
[38]LIU L,YIN K K,VAN DE PANNE M,et al.Sampling-based contact-rich motion control[C]//ACM Transactions on Gra-phics (TOG).New York,USA,2010,29(4):128.
[39]MIN J,CHAI J.Motion graphs++:a compact generative model for semantic motion analysis and synthesis[J].ACM Transactions on Graphics (TOG),2012,31(6):153.
[40]HWANG J,KIM J,SUH I H,et al.Real-time Locomotion Controller using an Inverted-Pendulum-based Abstract Model[J].Computer Graphics Forum,2018,37(2):287-296.
[41]RAJESWARAN A,GHOTRA S,RAVINDRAN B,et al.Epopt:Learning robust neural network policies using model ensembles[J].arXiv:1610.01283,2016.
[42]SU W,DU D,YANG X,et al.Interactive sketch-based normal map generation with deep neural networks[J].Proceedings of the ACM on Computer Graphics and Interactive Techniques,2018,1(1):22.
[43]LEVINE S,WANG J M,HARAUX A,et al.Continuous character control with low-dimensional embeddings[J].ACM Tran-sactions on Graphics (TOG),2012,31(4):28.
[44]CLEGG A,YU W,TAN J,et al.Learning to dress:synthesizing human dressing motion via deep reinforcement learning[C]//SIGGRAPH Asia 2018 Technical Papers.Tokyo,Japan,2018:179.
[45]PENG X B,BERSETH G,VAN DE PANNE M.Terrain-adaptive locomotion skills using deep reinforcement learning[J].ACM Transactions on Graphics (TOG),2016,35(4):81.
[46]LIU L,HODGINS J.Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning[J].ACM Transactions on Graphics (TOG),2018,37(4):142.
[47]NAKADA M,CHEN H,TERZOPOULOS D.Learning Biomi-metic Perception for Human Sensorimotor Control[C]//Procee-dings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Salt Lake City,USA,2018:1917-1922.
[48]NAKADA M,ZHOU T,CHEN H,et al.Deep learning of biomimetic sensorimotor control for biomechanical human animation[J].ACM Transactions on Graphics (TOG),2018,37(4):56.
[49]NAIR A,MCGREW B,ANDRYCHOWICZ M,et al.Overco-ming exploration in reinforcement learning with demonstrations[C]//2018 IEEE International Conference on Robotics and Automation (ICRA).IEEE,Brisbane,Australis,2018:6292-6299.
[50]GUPTA A,SATKIN S,EFROS A A,et al.From 3d scene geometry to human workspace[C]//2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2011:1961-1968.
[51]KIM V G,CHAUDHURI S,GUIBAS L,et al.Shape2pose:Human-centric shape analysis[J].ACM Transactions on Graphics (TOG),2014,33(4):120.
[52]SAVVA M,CHANG A X,Hanrahan P,et al.PiGraphs:learning interaction snapshots from observations[J].ACM Transactions on Graphics (TOG),2016,35(4):139.
[53]KAPADIA M,XU X H,NITTI M,et al.Precision:Precompu-ting environment semantics for contact-rich character animation[C]//Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games.ACM,2016:29-37.
[54]CHANG X,HOSPEDALES T M,XIANG T.Multi-level factorisation net for person re-identification[C]//CVPR.Salt Lake City,USA,2018:2.
[55]KULKARNI T D,WHITNEY W F,KOHLI P,et al.Deep convolutional inverse graphics network[C]//Advances in Neural Information Processing Systems.Long Beach,USA,2015:2539-2547.
[56]CHOI B,LEWIS J P,SEOL Y,et al.SketchiMo:sketch-basedmotion editing for articulated characters[J].ACM Transactions on Graphics (TOG),2016,35(4):146.
[57]NEFF M,FIUME E.Aesthetic edits for character animation[C]//Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation.San Diego,California:ACM,2003.
[58]COLEMAN P,BIBLIOWICZ J,SINGH K,et al.Staggered poses:a character motion representation for detail-preserving editing of pose and coordinated timing[C]//Eurographics/ACM SIGGRAPH Symposium on Computer Animation,SCA 2008.Dublin,Ireland,DBLP,2008:137-146.
[59]GUAY M,GLEICHER M,CANI M P.Adding dynamics tosketch-based character animations[C]//Sketch-Based Interfaces and Modeling.2015:27-34.
[60]IKEMOTO L,ARIKAN O,FORSYTH D.Generalizing motion edits with Gaussian processes[J].Acm Transactions on Graphi-cs,2009,28(1):1-12.
[61]HSU E,PULLI K.Style translation for human motion[J].ACM Transactions on Graphics (TOG),2005,24(3):1082-1089.
[62]GATYS L A,ECKER S,BETHGE M.A neural algorithm of artistic style[J].arXiv:1508.06576,2015.
[63]GATYS L A,ECKER A S,BETHGE M.Image style transfer using convolutional neural networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Las Vegas,USA,2016:2414-2423.
[64]BRAND M,HERTZMANN A.Style machines[C]//Procee-dings of the 27th annual conference on Computer graphics and interactive techniques.ACM Press/Addison-Wesley Publishing Co.,2000:183-192.
[65]MIN J,LIU H,CHAI J.Synthesis and editing of personalizedstylistic human motion[C]//Symposium on Interactive 3d Graphics(Si3d 2010).Washington,DC,USA,DBLP,2010:39-46.
[66]HOLDEN D,HABIBIE I,KUSAJIMA I,et al.Fast NeuralStyle Transfer for Motion Data[J].IEEE computer graphics and applications,2017,37(4):42-49.
[1] 余雪勇, 陈涛. 边缘计算场景中基于虚拟映射的隐私保护卸载算法[J]. 计算机科学, 2021, 48(1): 65-71.
[2] 单美静, 秦龙飞, 张会兵. L-YOLO:适用于车载边缘计算的实时交通标识检测模型[J]. 计算机科学, 2021, 48(1): 89-95.
[3] 何彦辉, 吴桂兴, 吴志强. 基于域适应的X光图像的目标检测[J]. 计算机科学, 2021, 48(1): 175-181.
[4] 李亚男, 胡宇佳, 甘伟, 朱敏. 基于深度学习的miRNA靶位点预测研究综述[J]. 计算机科学, 2021, 48(1): 209-216.
[5] 张艳梅, 楼胤成. 基于深度神经网络的庞氏骗局合约检测方法[J]. 计算机科学, 2021, 48(1): 273-279.
[6] 庄世杰, 於志勇, 郭文忠, 黄昉菀. 基于Zoneout的跨尺度循环神经网络及其在短期电力负荷预测中的应用[J]. 计算机科学, 2020, 47(9): 105-109.
[7] 张佳嘉, 张小洪. 多分支卷积神经网络肺结节分类方法及其可解释性[J]. 计算机科学, 2020, 47(9): 129-134.
[8] 朱玲莹, 桑庆兵, 顾婷婷. 基于视差信息的无参考立体图像质量评价[J]. 计算机科学, 2020, 47(9): 150-156.
[9] 赵钦炎, 李宗民, 刘玉杰, 李华. 基于信息熵的级联Siamese网络目标跟踪[J]. 计算机科学, 2020, 47(9): 157-162.
[10] 游兰, 韩雪薇, 何正伟, 肖丝雨, 何渡, 潘筱萌. 基于改进Seq2Seq的短时AIS轨迹序列预测模型[J]. 计算机科学, 2020, 47(9): 169-174.
[11] 崔彤彤, 王桂玲, 高晶. 基于1DCNN-LSTM的船舶轨迹分类方法[J]. 计算机科学, 2020, 47(9): 175-184.
[12] 刘海潮, 王莉. 基于深度图卷积胶囊网络的图分类模型[J]. 计算机科学, 2020, 47(9): 219-225.
[13] 郭杰, 高希然, 陈莉, 傅游, 刘颖. 用数据驱动的编程模型并行多重网格应用[J]. 计算机科学, 2020, 47(8): 32-40.
[14] 池昊宇, 陈长波. 基于神经网络的循环分块大小预测[J]. 计算机科学, 2020, 47(8): 62-70.
[15] 赵威, 林煜明, 王超强, 蔡国永. 基于依赖联系分析的观点词对协同抽取[J]. 计算机科学, 2020, 47(8): 164-170.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] 雷丽晖,王静. 可能性测度下的LTL模型检测并行化研究[J]. 计算机科学, 2018, 45(4): 71 -75 .
[2] 孙启,金燕,何琨,徐凌轩. 用于求解混合车辆路径问题的混合进化算法[J]. 计算机科学, 2018, 45(4): 76 -82 .
[3] 张佳男,肖鸣宇. 带权混合支配问题的近似算法研究[J]. 计算机科学, 2018, 45(4): 83 -88 .
[4] 伍建辉,黄中祥,李武,吴健辉,彭鑫,张生. 城市道路建设时序决策的鲁棒优化[J]. 计算机科学, 2018, 45(4): 89 -93 .
[5] 史雯隽,武继刚,罗裕春. 针对移动云计算任务迁移的快速高效调度算法[J]. 计算机科学, 2018, 45(4): 94 -99 .
[6] 周燕萍,业巧林. 基于L1-范数距离的最小二乘对支持向量机[J]. 计算机科学, 2018, 45(4): 100 -105 .
[7] 刘博艺,唐湘滟,程杰仁. 基于多生长时期模板匹配的玉米螟识别方法[J]. 计算机科学, 2018, 45(4): 106 -111 .
[8] 耿海军,施新刚,王之梁,尹霞,尹少平. 基于有向无环图的互联网域内节能路由算法[J]. 计算机科学, 2018, 45(4): 112 -116 .
[9] 崔琼,李建华,王宏,南明莉. 基于节点修复的网络化指挥信息系统弹性分析模型[J]. 计算机科学, 2018, 45(4): 117 -121 .
[10] 杨羽琦,章国安,金喜龙. 车载自组织网络中基于车辆密度的双簇头路由协议[J]. 计算机科学, 2018, 45(4): 126 -130 .