基于动态拓扑图的人体骨架动作识别算法

doi:10.11896/jsjkx.210900059

计算机科学 ›› 2022, Vol. 49 ›› Issue (2): 62-68.doi: 10.11896/jsjkx.210900059

• 计算机视觉:理论与应用 • 上一篇下一篇

基于动态拓扑图的人体骨架动作识别算法

解宇¹, 杨瑞玲¹, 刘公绪², 李德玉¹, 王文剑¹

1 山西大学计算机与信息技术学院太原030006
2 西安电子科技大学电子工程学院西安710071

收稿日期:2021-09-07 修回日期:2021-09-22 出版日期:2022-02-15 发布日期:2022-02-23
通讯作者: 王文剑(wjwang@sxu.edu.cn)
作者简介:yuxie@sxu.edu.cn
基金资助:
国家自然科学基金(62076154,62106131,62106134);中央引导地方科技发展资金项目(YDZX20201400001224);山西省国际科技合作计划项目(201903D421050)

Human Skeleton Action Recognition Algorithm Based on Dynamic Topological Graph

XIE Yu¹, YANG Rui-ling¹, LIU Gong-xu², LI De-yu¹, WANG Wen-jian¹

1 School of Computer and Information Technology,Shanxi University,Taiyuan 030006,China
2 School of Electronic Engineering,Xidian University,Xi'an 710071,China

Received:2021-09-07 Revised:2021-09-22 Online:2022-02-15 Published:2022-02-23
About author:XIE Yu,born in 1993,Ph.D,lecturer,is a member of China Computer Federation.His main research interests include graph neural networks and so on.
WANG Wen-jian,born in 1968,Ph.D,professor,Ph.D supervisor,is a distinguished member of China Computer Federation.Her main research interests include machine learning and neural networks.
Supported by:
National Natural Science Foundation of China(62076154,62106131,62106134),Program of Central Funds Guiding the Local Science and Technology Development(YDZX20201400001224) and Key R & D program of Shanxi Province(International Cooperation)(201903D421050).

摘要/Abstract

摘要： 传统的人体骨架动作识别算法采用手动构建拓扑图的方式来建模包含在多个视频帧中的动作序列,并针对性地学习每个视频帧以反映数据变化,这容易造成计算代价大、网络泛化性低和灾难性遗忘等问题。针对上述问题,提出了基于动态拓扑图的人体骨架动作识别算法,使用持续学习思想动态构建人体骨架拓扑图。将具有多关系特性的人体骨架序列数据重新编码为关系三元组,并基于长短期记忆网络,通过解耦合的方式学习特征嵌入。当处理新骨架关系三元组时,使用部分更新机制动态构建人体骨架拓扑图,并采用基于时空图卷积网络的骨架动作识别算法来实现动作识别。实验结果表明,所提方法在Kinetics-Skeleton,NTU-RGB+D(X-Sub)和NTU-RGB+D(X-View)基准数据集上分别取得了40%,85%和90%的识别准确率,提高了人体骨架动作识别的准确率。

关键词: 持续学习, 人体动作识别, 人体骨架数据, 图卷积网络, 灾难性遗忘

Abstract: Traditional human skeleton action recognition algorithms manually construct topological graphs to model the action sequence contained in multiple video frames and learn each video frame to reflect the data changes,which may lead to the high computational cost,low network generalization performance and catastrophic forgetting.To solve these problems,a human skeleton action recognition algorithm based on dynamic topological graph is proposed,in which the human skeleton topological graph is dynamically constructed based on continuous learning.Specifically,human skeleton sequence data with multi-relationship characte-ristics are recoded into relationship triplets,and feature embedding is learned in a decoupling manner via the long short-term me-mory network.When handling new skeleton relationship triplets,we dynamically construct the human skeleton topological graph by a partial update mechanism,and then send it to the skeleton action recognition algorithm based on spatio-temporal graph convolution network for action recognition.Experimental results demonstrate that the proposed algorithm achieves 40%,85% and 90% recognition accuracy on three benchmark datasets,namely Kinetics-Skeleton,NTU-RGB+D(X-Sub) and NTU-RGB+D(X-View),respectively,which improve the accuracy of human skeleton action recognition.

Key words: Catastrophic forgetting, Continual learning, Graph convolution network, Human action recognition, Human skeleton data

中图分类号:

TP391

解宇, 杨瑞玲, 刘公绪, 李德玉, 王文剑. 基于动态拓扑图的人体骨架动作识别算法[J]. 计算机科学, 2022, 49(2): 62-68. https://doi.org/10.11896/jsjkx.210900059

XIE Yu, YANG Rui-ling, LIU Gong-xu, LI De-yu, WANG Wen-jian. Human Skeleton Action Recognition Algorithm Based on Dynamic Topological Graph[J]. Computer Science, 2022, 49(2): 62-68. https://doi.org/10.11896/jsjkx.210900059

参考文献

[1]CHEN Y,TIAN Y,HE M.Monocular Human Pose Estimation:A Survey of Deep Learning-Based Methods[J].Computer Vision and Image Understanding,2020,192:102897.
[2]SONG S,LAN C,XING J,et al.An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data[C]//AAAI Conference on Artificial Intelligence.2017:4263-4270.
[3]YAN S,XIONG Y,LIN D.Spatial Temporal Graph Convolu-tional Networks for Skeleton-Based Action Recognition[C]//AAAI Conference on Artificial Intelligence.2018:7444-7452.
[4]XIONG X,MIN W,ZHENG W S,et al.S3D-CNN:Skeleton-Based 3D Consecutive-Low-Pooling Neural Network for Fall Detection[J].Applied Intelligence,2020,50(10):3521-3534.
[5]SHI L,ZHANG Y,CHENG J,et al.Two-Stream AdaptiveGraph Convolutional Networks for Skeleton-Based Action Re-cognition[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:12026-12035.
[6]ZHANG P,LAN C,ZENG W,et al.Semantics-Guided NeuralNetworks for Efficient Skeleton-Based Human Action Recognition[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:1112-1121.
[7]DING C Y,LIU K,LI G,et al.Spatio-Temporal Weighted Posture Motion Features for Human Skeleton Action Recognition Research[J].Chinese Journal of Computers,2020,43(1):29-40.
[8]TIAN Z Q,DENG C H,ZHANG J W.Human Behavior Recognition Algorithm Based on Skeletal Temporal Divergence Feature[J].Journal of Computer Applications,2021,41(5):1450-1457.
[9]SHI L,ZHANG Y,CHENG J,et al.Skeleton-based Action Re-cognition with Directed Graph Neural Networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.2019:7912-7921.
[10]TANG Y,TIAN Y,LU J,et al.Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:5323-5332.
[11]THAKKAR K,NARAYANAN P J.Part-Based Graph Convolutional Network for Action Recognition[J].arXiv:1809.04983,2018.
[12]LI M,CHEN S,CHEN X,et al.Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction[J/OL].IEEE Transactions on Pattern Ana-lysis and Machine Intelligence.https://ieeexplore.ieee.org/document/9334430.
[13]LI B,LI X,ZHANG Z,et al.Spatio-temporal Graph Routing for Skeleton-based Action Recognition[C]//AAAI Conference on Artificial Intelligence.2019:8561-8568.
[14]HADSELL R,RAO D,RUSU A A,et al.Embracing Change:Continual Learning in Deep Neural Networks[J].Trends in Cognitive Sciences,2020,24(12):1028-1040.
[15]CHEN P H,WEI W,HSIEH C,et al.Overcoming Catastrophic Forgetting by Generative Regularization[J].arXiv:1912.01238,2019.
[16]D'AUTUME C D M,RUDER S,KONG L,et al.EpisodicMemory in Lifelong Language Learning[J].Advances in Neural Information Processing Systems,2019,32:13143-13152.
[17]ROLNICK D, AHUJA A, SCHWARZ J,et al.Experience Replay for Continual Learning[J].Advances in Neural Information Processing Systems,2019,32:350-360.
[18]LIU L,PU H Y.Real-time LSTM-based Multi-dimensional Features Gesture Recognition[J].Computer Science,2021,48(8):328-333.
[19]KOU X,LIN Y,LIU S,et al.Disentangle-based ContinualGraph Representation Learning[C]//Conference on Empirical Methods in Natural Language Processing.2020:2961-2972.
[20]PENG W,HONG X,CHEN H,et al.Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching[C]//AAAI Conference on Artificial Intelligence.2020:2669-2676.
[21]PLIZZARI C,CANNICI M,MATTEUCCI M.Skeleton-basedAction Recognition Via Spatial and Temporal Transformer Networks[J].Computer Vision and Image Understanding,2021,208:103219.

相关文章 13

[1]	刘冬梅, 徐洋, 吴泽彬, 刘倩, 宋斌, 韦志辉. 基于边框距离度量的增量目标检测方法 Incremental Object Detection Method Based on Border Distance Measurement 计算机科学, 2022, 49(8): 136-142. https://doi.org/10.11896/jsjkx.220100132
[2]	汪鸣, 彭舰, 黄飞虎. 基于多时间尺度时空图网络的交通流量预测模型 Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction 计算机科学, 2022, 49(8): 40-48. https://doi.org/10.11896/jsjkx.220100188
[3]	李健智, 王红玲, 王中卿. 基于图卷积网络的专利摘要自动生成研究 Automatic Generation of Patent Summarization Based on Graph Convolution Network 计算机科学, 2022, 49(6A): 172-177. https://doi.org/10.11896/jsjkx.210400117
[4]	赵小虎, 叶圣, 李晓. 多算法融合的骨骼重建信息动作分类方法 Multi-algorithm Fusion Behavior Classification Method for Body Bone Information Reconstruction 计算机科学, 2022, 49(6): 269-275. https://doi.org/10.11896/jsjkx.210500070
[5]	周海榆, 张道强. 面向多中心数据的超图卷积神经网络及应用 Multi-site Hyper-graph Convolutional Neural Networks and Application 计算机科学, 2022, 49(3): 129-133. https://doi.org/10.11896/jsjkx.201100152
[6]	潘志豪, 曾碧, 廖文雄, 魏鹏飞, 文松. 基于交互注意力图卷积网络的方面情感分类 Interactive Attention Graph Convolutional Networks for Aspect-based Sentiment Classification 计算机科学, 2022, 49(3): 294-300. https://doi.org/10.11896/jsjkx.210100180
[7]	宋龙泽, 万怀宇, 郭晟楠, 林友芳. 面向出租车空载时间预测的多任务时空图卷积网络 Multi-task Spatial-Temporal Graph Convolutional Network for Taxi Idle Time Prediction 计算机科学, 2021, 48(7): 112-117. https://doi.org/10.11896/jsjkx.201000089
[8]	程思伟, 葛唯益, 王羽, 徐建. BGCN:基于BERT和图卷积网络的触发词检测 BGCN:Trigger Detection Based on BERT and Graph Convolution Network 计算机科学, 2021, 48(7): 292-298. https://doi.org/10.11896/jsjkx.200500133
[9]	宋元隆, 吕光宏, 王桂芝, 贾吾财. 基于图卷积神经网络的SDN网络流量预测 SDN Traffic Prediction Based on Graph Convolutional Network 计算机科学, 2021, 48(6A): 392-397. https://doi.org/10.11896/jsjkx.200800090
[10]	吕明琪, 洪照雄, 陈铁明. 一种融合时空关联与社会事件的交通流预测方法 Traffic Flow Forecasting Method Combining Spatio-Temporal Correlations and Social Events 计算机科学, 2021, 48(2): 264-270. https://doi.org/10.11896/jsjkx.200300098
[11]	叶松涛, 周扬正, 范红杰, 陈正雷. 融合因果关系和时空图卷积网络的人体动作识别 Joint Learning of Causality and Spatio-Temporal Graph Convolutional Network for Skeleton- based Action Recognition 计算机科学, 2021, 48(11A): 130-135. https://doi.org/10.11896/jsjkx.201200205
[12]	洪耀球. 基于深度信念网络的视觉人体动作识别 Visual Human Action Recognition Based on Deep Belief Network 计算机科学, 2021, 48(11A): 400-403. https://doi.org/10.11896/jsjkx.210200079
[13]	蒋宗礼, 李苗苗, 张津丽. 基于融合元路径图卷积的异质网络表示学习 Graph Convolution of Fusion Meta-path Based Heterogeneous Network Representation Learning 计算机科学, 2020, 47(7): 231-235. https://doi.org/10.11896/jsjkx.190600085

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于动态拓扑图的人体骨架动作识别算法

Human Skeleton Action Recognition Algorithm Based on Dynamic Topological Graph

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 13

Metrics

本文评价

推荐阅读 0