计算机科学 ›› 2020, Vol. 47 ›› Issue (6A): 139-147.doi: 10.11896/JsJkx.190900176
赫磊, 邵展鹏, 张剑华, 周小龙
HE Lei, SHAO Zhan-peng, ZHANG Jian-hua and ZHOU Xiao-long
摘要： 行为识别是计算机视觉领域的基本问题之一,基于深度学习的行为识别算法是当前行为识别的主流算法。在已有的研究中,传统特征提取方法一般是通过人工观察和设计,手动设计出能够表征视频动作的特征。然而,在手工特征表达的基础上构建复杂分类模型的方法已经不能适应高识别精度和应用性的要求,而深度学习的引入为行为识别带来了新的发展方向。文中主要综述了基于深度学习的行为识别算法,首先介绍了行为识别的研究背景和意义,并分别对行为识别的传统学习方法和深度学习方法进行了介绍;然后对深度学习下的算法模型结构进行分类介绍,包括Two-Stream、3D-ConvNet、融合CNN-LSTM 3种算法模型结构;最后介绍了目前常用的公开验证数据集,并主要针对基于两种数据模态的识别算法进行了横向比较,一种是基于RGB视频的UCF101和HMDB51数据集,一种是基于人体骨架序列视频的NTU RGB＋D数据集。实验结果表明:深度学习方法已经取得了很大的进步,卷积神经网络的应用极大地促进了行为识别算法的发展,逐步替代了基于手工提取特征的传统方法,尤其采用了卷积神经网络算法之后在行为数据集上的准确率有了显著提高。对于RGB视频而言,Two-Stream和3DConvNet是算法模型结构的主流,对于骨架序列视频而言,Two-Stream和融合时空图模型是算法模型结构的主流。
中图分类号:
| [1] WANG X.Intelligent multi-camera video surveillance:A review.Pattern Recognition Letters,2013,34(1):3-19. [2] TURAGA P,CHELLAPPA R,SUBRAHMANIAN V S,et al.Machine Recognition of Human Activities:A Survey.IEEE Trans.Circuits Syst.Video Technol.,2008,18(11):1473-1488. [3] ELLIS C,MASOOD S Z,TAPPEN M F,et al.Exploring the trade-off between accuracy and observational latency in action recognition.Int.J.Comput.Vis.,2013,101(3):420-436. [4] ZHANG W,SMITH M L,SMITH L N,et al.Gender and gaze gesture recognition for human-computer interaction//Computer Vision and Image Understanding.2016:32-50. [5] TAKANO W,NAKAMURA Y.Statistical mutual conversion between whole body motion primitives and linguistic sentences for human motions.Int.J.Rob.Res.,2015,34(10):1314-1328. [6] CALO R.Robotics and the Lessons of Cyberlaw.California Law Review,2014,103(3). [7] CAMPORESI C,KALLMANN M,HAN J J,et al.VR solutions for improving physical therapy//IEEE Virtual Reality Conference.2013:77-78. [8] CHAO M W,LIN C H,ASSA J,et al.Human motion retrieval from hand-drawn sketch.IEEE Trans.Vis.Comput.Graph.,2012,18(5):729-740. [9] KRIZHEVSKY A,SUTSKEVER I,HINTON G.Imagenet classification with deep convolutional neural networks//Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).2012:1097-1105. [10] GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate obJect detection and semantic segmentation//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014:580-587. [11] FARABET C,COUPRIE C,LECUN Y.Learning hierarchicalfeatures for scene labeling,IEEE Trans.Pattern Anal.Mach.Intell.,2013,35(8):1915-1929. [12] SU S,LIU Z,XU S,et al.Sparse auto-encoder based feature learning for human body detection indepth image.Signal Processing,2015,112(1):43-52. [13] VIEIRA A W,NASCIMENTO E R,OLIVEIRA G L,et al.STOP:Space-Time Occupancy Patterns for 3D Action Recognition from Depth Map Sequences//Iberoamerican Congress on Pattern Recognition.2012:252-259. [14] SEVILLALARA L,LIAO Y,GUNEY F,et al.On the Integration of Optical Flow and Action Recognition//German Conference on Pattern Recognition.2018:281-297. [15] HE K,ZHANG X,REN S,et al.Deep Residual Learning for Image Recognition//Computer Vision and Pattern Recognition.2016:770-778. [16] TSENG H C,SHYU J J,CHANG J Y,et al.Exploiting Automatic Image Segmentation to Human Detection and Depth Estimation///Proc of the IEEE Symposium on Computational Intelligence for Multimedia,Signal and Vision Processing.Paris,France,2011:19-25. [17] KIM W H,JEONG T I,KIM J N.Video Segmentation Algorithm Using Threshold and Weighting Based on Moving Sliding Window//Proc of the 11th International Conference on Advanced Communication Technology.Pyeongchang County,Repulic of Korea,2009:1781-1784 [18] SALMANE H,RUICHEK Y,KHOUDOUR L.ObJect Tracking Using Harris Corner Points Based Optical Flow Propagation and Kalman filter//Proc of the 14th International IEEE Confe-rence on Intelligent Transportation Systems.Washington,USA,2011:67-73. [19] YANG J,XU Y S,CHEN C S.Hidden Markov Model Approach to Skill Learning and Its Application to Telerobotics.IEEE Trans on Robotics and Automation,1994,10(5):621-631. [20] BOBICK A,DAVIS J.An appearance-based representation of action//Proceedings of the 13th International Conference on Pattern Recognition.Vienna:IEEE,1996:307-312. [21] WEINLAND D,RONFARD R,BOYER E.Free viewpoint action recognition using motion history volumes.Computer Vision and Image Understanding,2006,104(2/3):249-257. [22] DOLLAR P,RABAUD V,COTTRELL G W,et al.Behavior recognition via sparse spatio-temporal features//International Conference on Computer Communications and Networks.2005:65-72. [23] LAPTEV I.On space-time interest points .International Journal of Computer Vision,2005,64 (2/3):10-123. [24] WONG S,CIPOLLA R.Extracting Spatiotemporal Interest Points using Global Information//International Conference on Computer Vision.2007:1-8. [25] WANG H,ULLAH M M,KLASER A,et al.Evaluation of local spatio-temporal features for action recognition//British Machine Vision Conference.2009:1-11. [26] WANG H,KLASER A,SCHMID C,et al.DensetraJectories and motion boundary descriptors foraction recognition.International Journal of Computer Vision,2013,103(1):60-79. [27] DOLLAR P,RABAUD V,COTTRELL G W,et al.Behavior recognition via sparse spatio-temporal features//International Conference on Computer Communications and Networks.2005:65-72. [28] WANG H,KLASER A,SCHMID C,et al.Dense traJectories and motion boundary descriptors for action recognition.International Journal of Computer Vision,2013,103(1):60-79. [29] NGUYEN T PMANZANERA A.Action recognition using bag of features extracted from a beam of traJectories//2013 IEEE International Conference on Image Processing.Melbourne,VIC,2013:4354-4357. [30] WANG H,SCHMID C.Action Recognition with Improved TraJectories//International Conference on Computer Vision.2013:3551-3558. [31] SHI C,WANG Y,JIA F,et al.Fisher vector for scene character recognition:A comprehensive evaluation.Pattern Recognition,2017,2017(72):1-14. [32] DANAFAR S,GHEISSARI N.Action recognition for surveillance applications using optic flow and SVM//AsianConfe-rence on Computer Vision.2007:457-466. [33] WANG Y,XU W.Leveraging deep learning with LDA-basedtext analytics to detect automobile insurance fraud.Decision Support Systems,2018,105:87-95. [34] IJJINA E P,MOHAN C K.Hybrid deep neural network model for human action recognition.Applied Soft Computing,2016,46:936-952. [35] KRIZHEVSKY A,SUTSKEVER I,HINTON G E,et al.ImageNet Classification with Deep Convolutional Neural Networks.Neural Information Processing Systems,2012,141(5):1097-1105. [36] GREFF K,SRIVASTAVA R K,KOUTNIK J,et al.LSTM:A Search Space Odyssey.IEEE Transactions on Neural Networks,2017,28(10):2222-2232. [37] COLLOBERT R,WESTON J,BOTTOU L,et al.Natural Language Processing (Almost) from Scratch.arXiv:1103.0398. [38] TARWANI K M,EDEM S.Survey on Recurrent Neural Network in Natural Language Processing.International Journal of Engineering Trends and Technology,2017,48(6):301-304. [39] WANG P,LI Z,HOU Y,et al.Action Recognition Based on Joint TraJectory Maps Using Convolutional Neural Networks//ACM Multimedia.2016:102-106. [40] LI C,HOU Y,WANG P,et al.Joint Distance Maps Based Action Recognition With Convolutional Neural Networks.IEEE Signal Processing Letters,2017,24(5):624-628. [41] WANG X,GAO L,WANG P,et al.Two-Stream 3-D convNetFusion for Action Recognition in Videos With Arbitrary Size and Length.IEEE Transactions on Multimedia,2018,20(3):634-644. [42] HOCHREITER S,SCHMIDHUBER J.Long short-term memo-ry.Neural Computation,1997,9(8):1735-1780. [43] DONAHUE J,HENDRICKS L A,GUADARRAMA S,et al.Long-term recurrent convolutional networks for visual recognition and description//Computer Vision and Pattern Recognition,2015.2625-2634. [44] KE Q,BENNAMOUN M,AN S,et al.A New Representation of Skeleton Sequences for 3D Action Recognition//Computer Vision and Pattern Recognition.2017:4570-4579. [45] SHAO Z,LI Y,GUO Y,et al.A Hierarchical Model for Action Recognition Based on Body Parts//2018 IEEE International Conference on Robotics and Automation (ICRA).Brisbane,QLD,2018:1978-1985. [46] SIMONYAN K,ZISSERMAN A.Two-Stream ConvolutionalNetworks for Action Recognition in Videos.arXiv:1406.2199. [47] FEICHTENHOFER C,PINZ A,ZISSERMAN A,et al.Convolutional Two-Stream Network Fusion for Video Action Recognition//Computer Vision and Pattern Recognition.2016:1933-1941. [48] WANG L,XIONG Y,WANG Z,et al.Towards Good Practices for Very Deep Two-Stream ConvNets.arXiv:1507.02159,2015. [49] SZEGEDY C,LIU W,JIA Y,et al.Going deeper with convolutions//Computer Vision and Pattern Recognition.2015:1-9. [50] SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition//International Conference on Learning Representations.2015. [51] WANG L,XIONG Y,WANG Z,et al.Temporal Segment Networks:Towards Good Practices for Deep Action Recognition//European Conference on Computer Vision.2016:20-36. [52] HE K,ZHANG X,REN S,et al.Deep Residual Learning forImage Recognition//Computer Vision and Pattern Recognition.2016:770-778. [53] FEICHTENHOFER C,PINZ A,WILDES R P,et al.Spatiotemporal Residual Networks for Video Action Recognition//Neural Information Processing Systems.2016:3468-3476. [54] YANG M,JI S,XU W,et al.Detecting human actions in surveillance videos//Proceedings of the TREC Video Retrieval Evaluation Workshop.2009. [55] BACCOUCHE M,MAMALET F,WOLF C,et al.Sequential deep learning for human action recognition//Human Beha-vior Unterstanding.2011:29-39. [56] JI S,XU W,YANG M,et al.3D Convolutional Neural Networks for Human Action Recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(1):221-231. [57] JI S,XU W,YANG M,et al.3D Convolutional Neural Networks for Human Action Recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(1):221-231. [58] TRAN D,BOURDEV L,FERGUS R,et al.Learning Spatiotemporal Features with 3D Convolutional Networks//International Conference on Computer Vision.2015:4489-4497. [59] TRAN D,RAY J,SHOU Z,et al.ConvNet Architecture Search for Spatiotemporal Feature Learning..arXiv:1708.05038,2017. [60] QIU Z,YAO T,MEI T,et al.Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks//International Conference on Computer Vision.2017:5534-5542. [61] SZEGEDY C,VANHOUCKE V,IOFFE S,et al.Rethinking the Inception Architecture for Computer Vision//Computer Vision and Pattern Recognition.2016:2818-2826. [62] DIBA A,FAYYAZ M,SHARMA V,et al.Temporal 3D ConvNets:New Architecture and Transfer Learning for Video Classification.arXiv:1711.08200,2017. [63] KIPF T,WELLING M.Semi-Supervised Classification with Graph Convolutional Networks//International Conference on Learning Representations.2017. [64] SHI L,ZHANG Y,CHENG J,et al.Non-Local Graph Convolutional Networks for Skeleton-Based Action Recognition.ar-Xiv:1805.07694v2. [65] KARPATHY A,TODERICI G,SHETTY S,et al.Large-Scale Video Classification with Convolutional Neural Networks//Computer Vision and Pattern Recognition.2014:1725-1732. [66] DONAHUE J,HENDRICKS L A,GUADARRAMA S,et al.Long-term recurrent convolutional networks for visual recognition and description//Computer Vision and Pattern Recognition.2015:2625-2634. [67] NG J Y,HAUSKNECHT M J,VIJAYANARASIMHAN S, et al.Beyond short snippets:Deep networks for video classification//Computer Vision and Pattern Recognition.2015:4694-4702. [68] SRIVASTAVA N,MANSIMOV E,SALAKHUDINOV R,et al.Unsupervised Learning of Video Representations using LSTMs//International Conference on Machine Learning.2015:843-852. [69] YAN S,XIONG Y,LIN D,et al.Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition//National Conference on Artificial Intelligence.2018:7444-7452. [70] SI C,JING Y,WANG W,et al.Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning//European Conference on Computer Vision.2018:106-121. [71] LI C,ZHONG Q,XIE D,et al.Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation//International Joint Conference on Artificial Intelligence.2018:786-792. [72] HASSNER T.A critical review of action recognition benchmarks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR).2013:245-250. [73] KARPATHY A,TODERICI G,SHETTY S,et al.Large-Scale Video Classification with Convolutional Neural Networks//Computer Vision and Pattern Recognition.2014:1725-1732. [74] SOOMRO K,ZAMIR A R,SHAH M,et al.UCF101:A Dataset of 101 Human Actions Classes From Videos in The Wild.arXiv:1212.0402,2012. [75] REDDY K K,SHAH M.Recognizing 50 human action categories of web videos.Machine Vision Applications,2013,24(5):971-981. [76] KUEHNE H,JHUANG H,GARROTE E,et al.HMDB:Alarge video database for human motion recognition//International Conference on Computer Vision.2011:2556-2563. [77] ZISSERMAN A,CARREIRA J,SIMONYAN K,et al.The Kinetics Human Action Video Dataset.arXiv:1705.06950,2017. [78] LI W,ZHANG Z,LIU Z,et al.Action recognition based on abag of 3D points//Computer Vision and Pattern Recognition.2010:9-14. [79] SHAHROUDY A,LIU J,NG T,et al.NTU RGB+D:A Large Scale Dataset for 3D Human Activity Analysis//Computer Vision and Pattern Recognition.2016:1010-1019. [80] TRAN D,WANG H,TORRESANI L,et al.A Closer Look at Spatiotemporal Convolutions for Action Recognition//Computer Vision and Pattern Recognition.2018:6450-6459. [81] ZHANG P,LAN C,XING J,et al.View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition//IEEE Transactions on Pattern Analysis and Machine Intelligence.2019:1-1. [82] SHI L,ZHANG Y,CHENG J,et al.Adaptive spectral graphconvolutional networks for skeleton-based action recognition.arXiv:1805.07694,2018. [83] SI C,CHEN W,WANG W,et al.An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Re-cognition.arXiv:1902.09130,2019. |
| [1] | 周乐员, 张剑华, 袁甜甜, 陈胜勇. 多层注意力机制融合的序列到序列中国连续手语识别和翻译 Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion 计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026 |
| [2] | 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204 |
| [3] | 饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277 |
| [4] | 汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108 |
| [5] | 李宗民, 张玉鹏, 刘玉杰, 李华. 基于可变形图卷积的点云表征学习 Deformable Graph Convolutional Networks Based Point Cloud Representation Learning 计算机科学, 2022, 49(8): 273-278. https://doi.org/10.11896/jsjkx.210900023 |
| [6] | 王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099 |
| [7] | 郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077 |
| [8] | 姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046 |
| [9] | 陈泳全, 姜瑛. 基于卷积神经网络的APP用户行为分析方法 Analysis Method of APP User Behavior Based on Convolutional Neural Network 计算机科学, 2022, 49(8): 78-85. https://doi.org/10.11896/jsjkx.210700121 |
| [10] | 朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥. 基于注意力机制的医学影像深度哈希检索算法 Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism 计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153 |
| [11] | 孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061 |
| [12] | 檀莹莹, 王俊丽, 张超波. 基于图卷积神经网络的文本分类方法研究综述 Review of Text Classification Methods Based on Graph Convolutional Network 计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064 |
| [13] | 胡艳羽, 赵龙, 董祥军. 一种用于癌症分类的两阶段深度特征选择提取算法 Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification 计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092 |
| [14] | 张颖涛, 张杰, 张睿, 张文强. 全局信息引导的真实图像风格迁移 Photorealistic Style Transfer Guided by Global Information 计算机科学, 2022, 49(7): 100-105. https://doi.org/10.11896/jsjkx.210600036 |
| [15] | 戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮. 基于DNGAN的磁共振图像超分辨率重建算法 Super-resolution Reconstruction of MRI Based on DNGAN 计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105 |
|
||
首页