Computer Science ›› 2020, Vol. 47 ›› Issue (6A): 139-147.doi: 10.11896/JsJkx.190900176

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Review of Deep Learning-based Action Recognition Algorithms

HE Lei, SHAO Zhan-peng, ZHANG Jian-hua and ZHOU Xiao-long   

  1. College of Computer Science and Technology,ZheJiang University of Technology,Hangzhou 310023,China
  • Published:2020-07-07
  • About author:HE Lei, born in 1994, master.His main research interests include image processing and action recognition.
    SHAO Zhan-peng, Ph.D, is a member of China Computer Federation.His research interests include action recognition and pose estimation.
  • Supported by:
    This work was supported the National Natural Science Foundation of China (20160283,61603341) and Natural Science Foundation of ZheJiang Province,China (KYY-ZX-20190013,KYY-ZX-20180114).

Abstract: Action recognition is one of the fundamental problems in the field of computer vision.Currently,deep learning-based method is one of the mainstream methods for action recognition.In the existing researches,the traditional feature extraction method generally manually designs features that can represent video actions.However,this method usually requires a particular model to classify features,which cannot achieve high performance in real applications,while the introduction of deep learning brings a new development direction for action recognition.This paper briefly reviews on the action recognition methods based on deep learning.Firstly,the research background and significance of action recognition are introduced,and the traditional methods and deep learning-based methods are surveyed respectively.Then,the model architectures of three algorithms based on deep learning are classified and introduced,namely Two-Stream network,3DConvNet,CNN-LSTM network.Finally,the common used public validation datasets are introduced,and horizontal comparison is carried out on the recognition algorithms based on two data modes.Among these datasets,they can be grouped into two categories,RGB-based (e.g.,UCF101,HMDB51) and skeleton-based datasets (e.g.,NTU RGB+D).Experimental results show that the deep learning-based methods have made great advances,and the application of convolutional neural network has greatly promoted the development of action recognition algorithm.They gradually replace the traditional method based on manual features extraction.For RGB-based action recognition,Two-Stream and 3DConvNet are currently state-of-the-art methods.For skeleton-based action recognition,Two-Stream and spatiotemporal graph network achieve the best performance.

Key words: Action recognition, Deep learning, Convolutional neural network, Recurrent neural network, 3D-ConvNet

CLC Number: 

  • TP391.4
[1] WANG X.Intelligent multi-camera video surveillance:A review.Pattern Recognition Letters,2013,34(1):3-19.
[2] TURAGA P,CHELLAPPA R,SUBRAHMANIAN V S,et al.Machine Recognition of Human Activities:A Survey.IEEE Trans.Circuits Syst.Video Technol.,2008,18(11):1473-1488.
[3] ELLIS C,MASOOD S Z,TAPPEN M F,et al.Exploring the trade-off between accuracy and observational latency in action recognition.Int.J.Comput.Vis.,2013,101(3):420-436.
[4] ZHANG W,SMITH M L,SMITH L N,et al.Gender and gaze gesture recognition for human-computer interaction//Computer Vision and Image Understanding.2016:32-50.
[5] TAKANO W,NAKAMURA Y.Statistical mutual conversion between whole body motion primitives and linguistic sentences for human motions.Int.J.Rob.Res.,2015,34(10):1314-1328.
[6] CALO R.Robotics and the Lessons of Cyberlaw.California Law Review,2014,103(3).
[7] CAMPORESI C,KALLMANN M,HAN J J,et al.VR solutions for improving physical therapy//IEEE Virtual Reality Conference.2013:77-78.
[8] CHAO M W,LIN C H,ASSA J,et al.Human motion retrieval from hand-drawn sketch.IEEE Trans.Vis.Comput.Graph.,2012,18(5):729-740.
[9] KRIZHEVSKY A,SUTSKEVER I,HINTON G.Imagenet classification with deep convolutional neural networks//Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).2012:1097-1105.
[10] GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate obJect detection and semantic segmentation//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014:580-587.
[11] FARABET C,COUPRIE C,LECUN Y.Learning hierarchicalfeatures for scene labeling,IEEE Trans.Pattern Anal.Mach.Intell.,2013,35(8):1915-1929.
[12] SU S,LIU Z,XU S,et al.Sparse auto-encoder based feature learning for human body detection indepth image.Signal Processing,2015,112(1):43-52.
[13] VIEIRA A W,NASCIMENTO E R,OLIVEIRA G L,et al.STOP:Space-Time Occupancy Patterns for 3D Action Recognition from Depth Map Sequences//Iberoamerican Congress on Pattern Recognition.2012:252-259.
[14] SEVILLALARA L,LIAO Y,GUNEY F,et al.On the Integration of Optical Flow and Action Recognition//German Conference on Pattern Recognition.2018:281-297.
[15] HE K,ZHANG X,REN S,et al.Deep Residual Learning for Image Recognition//Computer Vision and Pattern Recognition.2016:770-778.
[16] TSENG H C,SHYU J J,CHANG J Y,et al.Exploiting Automatic Image Segmentation to Human Detection and Depth Estimation///Proc of the IEEE Symposium on Computational Intelligence for Multimedia,Signal and Vision Processing.Paris,France,2011:19-25.
[17] KIM W H,JEONG T I,KIM J N.Video Segmentation Algorithm Using Threshold and Weighting Based on Moving Sliding Window//Proc of the 11th International Conference on Advanced Communication Technology.Pyeongchang County,Repulic of Korea,2009:1781-1784
[18] SALMANE H,RUICHEK Y,KHOUDOUR L.ObJect Tracking Using Harris Corner Points Based Optical Flow Propagation and Kalman filter//Proc of the 14th International IEEE Confe-rence on Intelligent Transportation Systems.Washington,USA,2011:67-73.
[19] YANG J,XU Y S,CHEN C S.Hidden Markov Model Approach to Skill Learning and Its Application to Telerobotics.IEEE Trans on Robotics and Automation,1994,10(5):621-631.
[20] BOBICK A,DAVIS J.An appearance-based representation of action//Proceedings of the 13th International Conference on Pattern Recognition.Vienna:IEEE,1996:307-312.
[21] WEINLAND D,RONFARD R,BOYER E.Free viewpoint action recognition using motion history volumes.Computer Vision and Image Understanding,2006,104(2/3):249-257.
[22] DOLLAR P,RABAUD V,COTTRELL G W,et al.Behavior recognition via sparse spatio-temporal features//International Conference on Computer Communications and Networks.2005:65-72.
[23] LAPTEV I.On space-time interest points .International Journal of Computer Vision,2005,64 (2/3):10-123.
[24] WONG S,CIPOLLA R.Extracting Spatiotemporal Interest Points using Global Information//International Conference on Computer Vision.2007:1-8.
[25] WANG H,ULLAH M M,KLASER A,et al.Evaluation of local spatio-temporal features for action recognition//British Machine Vision Conference.2009:1-11.
[26] WANG H,KLASER A,SCHMID C,et al.DensetraJectories and motion boundary descriptors foraction recognition.International Journal of Computer Vision,2013,103(1):60-79.
[27] DOLLAR P,RABAUD V,COTTRELL G W,et al.Behavior recognition via sparse spatio-temporal features//International Conference on Computer Communications and Networks.2005:65-72.
[28] WANG H,KLASER A,SCHMID C,et al.Dense traJectories and motion boundary descriptors for action recognition.International Journal of Computer Vision,2013,103(1):60-79.
[29] NGUYEN T PMANZANERA A.Action recognition using bag of features extracted from a beam of traJectories//2013 IEEE International Conference on Image Processing.Melbourne,VIC,2013:4354-4357.
[30] WANG H,SCHMID C.Action Recognition with Improved TraJectories//International Conference on Computer Vision.2013:3551-3558.
[31] SHI C,WANG Y,JIA F,et al.Fisher vector for scene character recognition:A comprehensive evaluation.Pattern Recognition,2017,2017(72):1-14.
[32] DANAFAR S,GHEISSARI N.Action recognition for surveillance applications using optic flow and SVM//AsianConfe-rence on Computer Vision.2007:457-466.
[33] WANG Y,XU W.Leveraging deep learning with LDA-basedtext analytics to detect automobile insurance fraud.Decision Support Systems,2018,105:87-95.
[34] IJJINA E P,MOHAN C K.Hybrid deep neural network model for human action recognition.Applied Soft Computing,2016,46:936-952.
[35] KRIZHEVSKY A,SUTSKEVER I,HINTON G E,et al.ImageNet Classification with Deep Convolutional Neural Networks.Neural Information Processing Systems,2012,141(5):1097-1105.
[36] GREFF K,SRIVASTAVA R K,KOUTNIK J,et al.LSTM:A Search Space Odyssey.IEEE Transactions on Neural Networks,2017,28(10):2222-2232.
[37] COLLOBERT R,WESTON J,BOTTOU L,et al.Natural Language Processing (Almost) from Scratch.arXiv:1103.0398.
[38] TARWANI K M,EDEM S.Survey on Recurrent Neural Network in Natural Language Processing.International Journal of Engineering Trends and Technology,2017,48(6):301-304.
[39] WANG P,LI Z,HOU Y,et al.Action Recognition Based on Joint TraJectory Maps Using Convolutional Neural Networks//ACM Multimedia.2016:102-106.
[40] LI C,HOU Y,WANG P,et al.Joint Distance Maps Based Action Recognition With Convolutional Neural Networks.IEEE Signal Processing Letters,2017,24(5):624-628.
[41] WANG X,GAO L,WANG P,et al.Two-Stream 3-D convNetFusion for Action Recognition in Videos With Arbitrary Size and Length.IEEE Transactions on Multimedia,2018,20(3):634-644.
[42] HOCHREITER S,SCHMIDHUBER J.Long short-term memo-ry.Neural Computation,1997,9(8):1735-1780.
[43] DONAHUE J,HENDRICKS L A,GUADARRAMA S,et al.Long-term recurrent convolutional networks for visual recognition and description//Computer Vision and Pattern Recognition,2015.2625-2634.
[44] KE Q,BENNAMOUN M,AN S,et al.A New Representation of Skeleton Sequences for 3D Action Recognition//Computer Vision and Pattern Recognition.2017:4570-4579.
[45] SHAO Z,LI Y,GUO Y,et al.A Hierarchical Model for Action Recognition Based on Body Parts//2018 IEEE International Conference on Robotics and Automation (ICRA).Brisbane,QLD,2018:1978-1985.
[46] SIMONYAN K,ZISSERMAN A.Two-Stream ConvolutionalNetworks for Action Recognition in Videos.arXiv:1406.2199.
[47] FEICHTENHOFER C,PINZ A,ZISSERMAN A,et al.Convolutional Two-Stream Network Fusion for Video Action Recognition//Computer Vision and Pattern Recognition.2016:1933-1941.
[48] WANG L,XIONG Y,WANG Z,et al.Towards Good Practices for Very Deep Two-Stream ConvNets.arXiv:1507.02159,2015.
[49] SZEGEDY C,LIU W,JIA Y,et al.Going deeper with convolutions//Computer Vision and Pattern Recognition.2015:1-9.
[50] SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition//International Conference on Learning Representations.2015.
[51] WANG L,XIONG Y,WANG Z,et al.Temporal Segment Networks:Towards Good Practices for Deep Action Recognition//European Conference on Computer Vision.2016:20-36.
[52] HE K,ZHANG X,REN S,et al.Deep Residual Learning forImage Recognition//Computer Vision and Pattern Recognition.2016:770-778.
[53] FEICHTENHOFER C,PINZ A,WILDES R P,et al.Spatiotemporal Residual Networks for Video Action Recognition//Neural Information Processing Systems.2016:3468-3476.
[54] YANG M,JI S,XU W,et al.Detecting human actions in surveillance videos//Proceedings of the TREC Video Retrieval Evaluation Workshop.2009.
[55] BACCOUCHE M,MAMALET F,WOLF C,et al.Sequential deep learning for human action recognition//Human Beha-vior Unterstanding.2011:29-39.
[56] JI S,XU W,YANG M,et al.3D Convolutional Neural Networks for Human Action Recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(1):221-231.
[57] JI S,XU W,YANG M,et al.3D Convolutional Neural Networks for Human Action Recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(1):221-231.
[58] TRAN D,BOURDEV L,FERGUS R,et al.Learning Spatiotemporal Features with 3D Convolutional Networks//International Conference on Computer Vision.2015:4489-4497.
[59] TRAN D,RAY J,SHOU Z,et al.ConvNet Architecture Search for Spatiotemporal Feature Learning..arXiv:1708.05038,2017.
[60] QIU Z,YAO T,MEI T,et al.Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks//International Conference on Computer Vision.2017:5534-5542.
[61] SZEGEDY C,VANHOUCKE V,IOFFE S,et al.Rethinking the Inception Architecture for Computer Vision//Computer Vision and Pattern Recognition.2016:2818-2826.
[62] DIBA A,FAYYAZ M,SHARMA V,et al.Temporal 3D ConvNets:New Architecture and Transfer Learning for Video Classification.arXiv:1711.08200,2017.
[63] KIPF T,WELLING M.Semi-Supervised Classification with Graph Convolutional Networks//International Conference on Learning Representations.2017.
[64] SHI L,ZHANG Y,CHENG J,et al.Non-Local Graph Convolutional Networks for Skeleton-Based Action
[65] KARPATHY A,TODERICI G,SHETTY S,et al.Large-Scale Video Classification with Convolutional Neural Networks//Computer Vision and Pattern Recognition.2014:1725-1732.
[66] DONAHUE J,HENDRICKS L A,GUADARRAMA S,et al.Long-term recurrent convolutional networks for visual recognition and description//Computer Vision and Pattern Recognition.2015:2625-2634.
[67] NG J Y,HAUSKNECHT M J,VIJAYANARASIMHAN S, et al.Beyond short snippets:Deep networks for video classification//Computer Vision and Pattern Recognition.2015:4694-4702.
[68] SRIVASTAVA N,MANSIMOV E,SALAKHUDINOV R,et al.Unsupervised Learning of Video Representations using LSTMs//International Conference on Machine Learning.2015:843-852.
[69] YAN S,XIONG Y,LIN D,et al.Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition//National Conference on Artificial Intelligence.2018:7444-7452.
[70] SI C,JING Y,WANG W,et al.Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning//European Conference on Computer Vision.2018:106-121.
[71] LI C,ZHONG Q,XIE D,et al.Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation//International Joint Conference on Artificial Intelligence.2018:786-792.
[72] HASSNER T.A critical review of action recognition benchmarks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR).2013:245-250.
[73] KARPATHY A,TODERICI G,SHETTY S,et al.Large-Scale Video Classification with Convolutional Neural Networks//Computer Vision and Pattern Recognition.2014:1725-1732.
[74] SOOMRO K,ZAMIR A R,SHAH M,et al.UCF101:A Dataset of 101 Human Actions Classes From Videos in The Wild.arXiv:1212.0402,2012.
[75] REDDY K K,SHAH M.Recognizing 50 human action categories of web videos.Machine Vision Applications,2013,24(5):971-981.
[76] KUEHNE H,JHUANG H,GARROTE E,et al.HMDB:Alarge video database for human motion recognition//International Conference on Computer Vision.2011:2556-2563.
[77] ZISSERMAN A,CARREIRA J,SIMONYAN K,et al.The Kinetics Human Action Video Dataset.arXiv:1705.06950,2017.
[78] LI W,ZHANG Z,LIU Z,et al.Action recognition based on abag of 3D points//Computer Vision and Pattern Recognition.2010:9-14.
[79] SHAHROUDY A,LIU J,NG T,et al.NTU RGB+D:A Large Scale Dataset for 3D Human Activity Analysis//Computer Vision and Pattern Recognition.2016:1010-1019.
[80] TRAN D,WANG H,TORRESANI L,et al.A Closer Look at Spatiotemporal Convolutions for Action Recognition//Computer Vision and Pattern Recognition.2018:6450-6459.
[81] ZHANG P,LAN C,XING J,et al.View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition//IEEE Transactions on Pattern Analysis and Machine Intelligence.2019:1-1.
[82] SHI L,ZHANG Y,CHENG J,et al.Adaptive spectral graphconvolutional networks for skeleton-based action recognition.arXiv:1805.07694,2018.
[83] SI C,CHEN W,WANG W,et al.An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Re-cognition.arXiv:1902.09130,2019.
[1] SHAN Mei-jing, QIN Long-fei, ZHANG Hui-bing. L-YOLO:Real Time Traffic Sign Detection Model for Vehicle Edge Computing [J]. Computer Science, 2021, 48(1): 89-95.
[2] HE Yan-hui, WU Gui-xing, WU Zhi-qiang. Domain Alignment Based Object Detection of X-ray Images [J]. Computer Science, 2021, 48(1): 175-181.
[3] LI Ya-nan, HU Yu-jia, GAN Wei, ZHU Min. Survey on Target Site Prediction of Human miRNA Based on Deep Learning [J]. Computer Science, 2021, 48(1): 209-216.
[4] WANG Rui-ping, JIA Zhen, LIU Chang, CHEN Ze-wei, LI Tian-rui. Deep Interest Factorization Machine Network Based on DeepFM [J]. Computer Science, 2021, 48(1): 226-232.
[5] YU Wen-jia, DING Shi-fei. Conditional Generative Adversarial Network Based on Self-attention Mechanism [J]. Computer Science, 2021, 48(1): 241-246.
[6] TONG Xin, WANG Bin-jun, WANG Run-zheng, PAN Xiao-qin. Survey on Adversarial Sample of Deep Learning Towards Natural Language Processing [J]. Computer Science, 2021, 48(1): 258-267.
[7] DING Yu, WEI Hao, PAN Zhi-song, LIU Xin. Survey of Network Representation Learning [J]. Computer Science, 2020, 47(9): 52-59.
[8] ZHUANG Shi-jie, YU Zhi-yong, GUO Wen-zhong, HUANG Fang-wan. Short Term Load Forecasting via Zoneout-based Multi-time Scale Recurrent Neural Network [J]. Computer Science, 2020, 47(9): 105-109.
[9] HE Xin, XU Juan, JIN Ying-ying. Action-related Network:Towards Modeling Complete Changeable Action [J]. Computer Science, 2020, 47(9): 123-128.
[10] ZHANG Jia-jia, ZHANG Xiao-hong. Multi-branch Convolutional Neural Network for Lung Nodule Classification and Its Interpretability [J]. Computer Science, 2020, 47(9): 129-134.
[11] YE Ya-nan, CHI Jing, YU Zhi-ping, ZHAN Yu-liand ZHANG Cai-ming. Expression Animation Synthesis Based on Improved CycleGan Model and Region Segmentation [J]. Computer Science, 2020, 47(9): 142-149.
[12] ZHU Ling-ying, SANG Qing-bing, GU Ting-ting. No-reference Stereo Image Quality Assessment Based on Disparity Information [J]. Computer Science, 2020, 47(9): 150-156.
[13] DENG Liang, XU Geng-lin, LI Meng-jie, CHEN Zhang-jin. Fast Face Recognition Based on Deep Learning and Multiple Hash Similarity Weighting [J]. Computer Science, 2020, 47(9): 163-168.
[14] YOU Lan, HAN Xue-wei, HE Zheng-wei, XIAO Si-yu, HE Du, PAN Xiao-meng. Improved Sequence-to-Sequence Model for Short-term Vessel Trajectory Prediction Using AIS Data Streams [J]. Computer Science, 2020, 47(9): 169-174.
[15] CUI Tong-tong, WANG Gui-ling, GAO Jing. Ship Trajectory Classification Method Based on 1DCNN-LSTM [J]. Computer Science, 2020, 47(9): 175-184.
Full text



[1] LEI Li-hui and WANG Jing. Parallelization of LTL Model Checking Based on Possibility Measure[J]. Computer Science, 2018, 45(4): 71 -75 .
[2] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[3] ZHANG Jia-nan and XIAO Ming-yu. Approximation Algorithm for Weighted Mixed Domination Problem[J]. Computer Science, 2018, 45(4): 83 -88 .
[4] WU Jian-hui, HUANG Zhong-xiang, LI Wu, WU Jian-hui, PENG Xin and ZHANG Sheng. Robustness Optimization of Sequence Decision in Urban Road Construction[J]. Computer Science, 2018, 45(4): 89 -93 .
[5] SHI Wen-jun, WU Ji-gang and LUO Yu-chun. Fast and Efficient Scheduling Algorithms for Mobile Cloud Offloading[J]. Computer Science, 2018, 45(4): 94 -99 .
[6] ZHOU Yan-ping and YE Qiao-lin. L1-norm Distance Based Least Squares Twin Support Vector Machine[J]. Computer Science, 2018, 45(4): 100 -105 .
[7] LIU Bo-yi, TANG Xiang-yan and CHENG Jie-ren. Recognition Method for Corn Borer Based on Templates Matching in Muliple Growth Periods[J]. Computer Science, 2018, 45(4): 106 -111 .
[8] GENG Hai-jun, SHI Xin-gang, WANG Zhi-liang, YIN Xia and YIN Shao-ping. Energy-efficient Intra-domain Routing Algorithm Based on Directed Acyclic Graph[J]. Computer Science, 2018, 45(4): 112 -116 .
[9] CUI Qiong, LI Jian-hua, WANG Hong and NAN Ming-li. Resilience Analysis Model of Networked Command Information System Based on Node Repairability[J]. Computer Science, 2018, 45(4): 117 -121 .
[10] WANG Zhen-chao, HOU Huan-huan and LIAN Rui. Path Optimization Scheme for Restraining Degree of Disorder in CMT[J]. Computer Science, 2018, 45(4): 122 -125 .