计算机科学 ›› 2021, Vol. 48 ›› Issue (8): 328-333.doi: 10.11896/jsjkx.210300079

• 人机交互 • 上一篇    下一篇

基于LSTM的多维度特征手势实时识别

刘亮, 蒲浩洋   

  1. 四川大学网络空间安全学院 成都610000
  • 收稿日期:2021-03-05 修回日期:2021-05-31 发布日期:2021-08-10
  • 通讯作者: 蒲浩洋(puhaoyang@outlook.com)
  • 基金资助:
    四川省科技计划资助(2021YFG0159)

Real-time LSTM-based Multi-dimensional Features Gesture Recognition

LIU Liang, PU Hao-yang   

  1. School of Cyber Science and Engineering,Sichuan University,Chengdu 610000,China
  • Received:2021-03-05 Revised:2021-05-31 Published:2021-08-10
  • About author:LIU Liang,born in 1982,Ph.D,is a member of China Computer Federation.His main research interests include vulnerability exploiting,malicious code analysis and hand gesture recognition.(liangzhai118@scu.edu.cn)PU Hao-yang,born in 2000,postgra-duate.His main research interests include vulnerability analysis and hand gesture recognition.
  • Supported by:
    Sichuan Science and Technology Program(2021YFG0159).

摘要: 手势识别广泛应用于传感领域,主要有基于计算机视觉、基于深度传感器与基于运动传感器等3种手势识别方式。基于运动传感器的手势识别具有输入数据少、速度快、直接获取手部三维信息的优点,逐渐成为当前的研究热点。传统基于运动传感器的手势识别本质为模式识别问题,其准确率严重依赖于先验经验提取的特征数据集。与传统的模式识别方法不同,深度学习可以在很大程度上减少人工启发式提取特征的工作量。为解决传统模式识别存在的问题,文中提出一种基于长短期记忆网络(LSTM)的多特征手势实时识别方法,通过充分的实验验证了该方法的性能。该方法首先定义了5种基本手势和7种复杂手势的手势库,基于手部姿态的运动学特征,进一步提取角度特征和位移特征,随后利用短时傅里叶变换(SFTF)提取传感器数据的频域特征,将3种特征输入深度神经网络LSTM中进行训练,从而对采集的手势进行分类识别。同时为了验证所提方法的有效性,通过自设计的手持式体验棒收集了6名志愿者的手势数据作为实验数据集。实验结果表明,提出的识别方法对于基本手势和复杂手势的识别准确率达到94.38%,与传统的支持向量机、K-近邻法和全连接神经网络相比,识别精度提升了近2%。

关键词: 动作捕捉, 惯性传感器, 人机交互, 手势识别

Abstract: Gesture recognition is widely used in the field of sensing.There are three kinds of gesture recognition methods based on computer vision,depth sensor and motion sensor.The recognition based on motion sensor has the advantages of less input data,high speed,and direct acquisition of hand 3D information,which has gradually become a research hotspot.Traditional gesture recognition based on motion sensor can be considered as a pattern recognition problem essentially and its accuracy depends heavily on feature data sets extracted from prior experience.Different from traditional pattern recognition methods,deep learning can greatly reduce the workload of artificial heuristic feature extraction.To solve the problem of traditional pattern recognition,this paper proposes a real-time multi-dimensional features recognition method based on Long Short-Term Memory(LSTM)and the performance of the method is verified by sufficient experiment.The method defines a gesture library consisting of five basic gestures and seven complex gestures at first.Based on the kinematic features of hand posture,the angle features and displacement features are extracted and then the frequency domain features of sensor data are extracted by short-time Fourier transform(SFTF).Then,three features are inputted into deep neural network LSTM for training,so the collected gestures are classified and recognized.At the same time,in order to verify the effectiveness of the proposed method,the gesture data of six volunteers are collected as the experimental data set by self-designed hand-held experience stick.The experimental results show that the accuracy of the recognition method proposed in this paper achieves 94.38% for basic and complex gestures,and the recognition accuracy is improved by nearly 2% compared with the traditional support vector machine,K-nearest neighbor method and fully connected neural network.

Key words: Gesture recognition, Human-computer interaction, Inertial sensor, Motion capture

中图分类号: 

  • TP391.41
[1]AKL A,FENG C,VALAEE S,et al.A Novel Accelerometer-Based Gesture Recognition System[J].IEEE Transactions on Signal Processing,2011,59(12):6197-6205.
[2]LU T.A motion control method of intelligent wheelchair based on hand gesture recognition[C]//2013 IEEE 8th Conference on Industrial Electronics and Applications.New York:IEEE,2013:957-962.
[3]ZHAO S Y,SHI J,DONG M,et al.Car control based on gesture recognition[J].Internal Combustion Engine and Accessories,2020(17):204-205.
[4]ZHOU J L,HE Y Z,ZHANG Y M,et al.Design and implementation of VR piano and gesture recognition based on Leapmotion[J].Electronic Testing,2020(21):5-9.
[5]WANG J S,CHUANG F C.An accelerometer-based digital pen with a trajectory recognition algorithm for handwritten digit and gesture recognition[J].IEEE Transactions on Industrial Electronics,2012,59(7):2998-3007.
[6]ZHANG X,CHEN X,LI Y,et al.A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors[J].IEEE Transactions on Systems,Man,and Cybernetics,2011,41(6):1064-1076.
[7]LU Z,CHEN X,LI Q,et al.A Hand Gesture RecognitionFramework and Wearable Gesture-Based Interaction Prototype for Mobile Devices[J].IEEE Transactions on Human-Machine Systems,2017,44(2):293-299.
[8]LIU Y H,SHENG L,ZHANG G L,et al.2D Human GestureTracking and Recognition by the Fusion of MEMS Inertial and Vision Sensors[J].IEEE Sensors Journal,2014,14(4):1160-1170.
[9]CHEN C,NASSER K,LIU K,et al.Fusion of Inertial andDepth Sensor Data for Robust Hand Gesture Recognition[J].IEEE Sensors Journal,2014,14(6):1898-1903.
[10]ZHAO N,YANG X D,ZHANG Z,et al.Circulating Nurse Assistant:Non-contact Body Centric Gesture Recognition towards Reducing Iatrogenic Contamination[J].IEEE Journal of Biomedical and Health Informatics,2020(25):2305-2316.
[11]OLIVER A,MAHESWARI N,SAMRAJ A,et al.Technology;New Findings from Helwan University Update Understanding of Technology (Smart Healthcare Solutions Using the Internet of Medical Things for Hand Gesture Recognition System)[J].Journal of Engineering,2020(99):1701-1706.
[12]CHEN L,LI Y F,LIU Z Y,et al.Gesture recognition in virtual reality interactive games [J].Technological Innovation and Application,2019(20):22-24.
[13]ZHU T,JIN G D,LU L B.Kinect Application Overview and Development Prospect[J].Modern Computer (Professional Edition),2013(6):8-11,33.
[14]CHEN L C,WANG F.A Survey on Hand Gesture Recognition[C]//2013 International Conference on Computer Sciences and Applications.Wuhan:IEEE,2013:313-316.
[15]MIAO Y W,LI J Y,LIU J Z,et al.Gesture recognition based on joint rotation feature and fingertip distance feature[J].Chinese Journal of Computers,2020,43(1):78-92.
[16]OSCAR D L,MIGUEL A L.A Survey on Human Activity Re-cognition using Wearable Sensors[J].IEEE Communications Surveys and Tutorials,2013,15(3):915-919.
[17]BENGIO Y.Deep learning of representations:Looking forward[C]//International Conference on Statistical Language and Speech Processing.Berlin:Springer,2013:1-37.
[18]YANG J B,NGUYEN M,LI L X,et al.Deep convolutional neural networks on multichannel time series for human activity re-cognition[C]//Twenty-Fourth International Joint Conference on Artificial Intelligence.2015:159-168.
[19]ALSHEIKH M A,SELIM A,LIN S,et al.Deep Activity Recognition Models with Triaxial Accelerometers[J].Computer Scien-ce,2015,1151:64-78.
[20]LANE N D,PETKO G,QENDRO L.DeepEar:robust smartphone audio sensing in unconstrained acoustic environments using deep learning[C]//Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing.New York:ACM,2015:283-294.
[21]THOMAS P,HAMMERLA N Y,OLIVIER P.Feature Lear-ning for Activity Recognition in Ubiquitous Computing[C]//Proceedings of the 22nd International Joint Conference on Artificial Intelligence(IJCAI 2011).Barcelona:DBLP,2011:215-226.
[22]VEPAKOMMA P,DE D,BHANSALI S,et al.A-Wristocracy:Deep Learning on Wrist-worn Sensing for Recognition of User Complex Activities[C]//IEEE Body Sensor Networks Confe-rence.New York:IEEE,2015:1-6.
[23]WALSE K H,DHARASKAR R V,THAKARE V M.PCABased Optimal ANN Classifiers for Human Activity Recognition Using Mobile Sensors Data[C]//International Conference on ICT for Intelligent Systems (ICTIS 2015).Cham:Springer,2016:429-436.
[24]HAMMERLAR N Y,HALLORAN S,PLOETZ T.Deep,Convo-lutional,and Recurrent Models for Human Activity Recognition using Wearables[J].Journal of Scientific Computing,2016,61(2):454-476.
[25]ALMASLUKH B,JALAL A.An effective deep autoencoder approach for online smartphone-based human activity recognition[C]//International Joint Conference on Computer Science.2017:160-165.
[26]WANG A,CHEN G,SHANG C,et al.Human Activity Recognition in a Smart Home Environment with Stacked Denoising Autoencoders[J].Springer,2016,9998:29-40.
[27]LI Y,SHI D,DING B,et al.Unsupervised Feature Learning for Human Activity Recognition Using Smartphone Sensors[C]//Mining Intelligence and Knowledge Exploration.2014,8891:99-107.
[28]XIONG Y,QUEK F.Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis[J].International Journal of Computer Vision,2006,69(3):353-371.
[1] 李荪, 曹峰.
智能语音技术端到端框架模型分析和趋势研究
Analysis and Trend Research of End-to-End Framework Model of Intelligent Speech Technology
计算机科学, 2022, 49(6A): 331-336. https://doi.org/10.11896/jsjkx.210500180
[2] 张继凯, 李琦, 王月明, 吕晓琪.
基于单目RGB图像的三维手势跟踪算法综述
Survey of 3D Gesture Tracking Algorithms Based on Monocular RGB Images
计算机科学, 2022, 49(4): 174-187. https://doi.org/10.11896/jsjkx.210700084
[3] 朱晨爽, 程时伟.
虚拟现实环境下基于眼动跟踪的导航需求预测与辅助
Prediction and Assistance of Navigation Demand Based on Eye Tracking in Virtual Reality Environment
计算机科学, 2021, 48(8): 315-321. https://doi.org/10.11896/jsjkx.200500031
[4] 王炽, 常俊.
基于3D卷积神经网络的CSI跨场景手势识别方法
CSI Cross-domain Gesture Recognition Method Based on 3D Convolutional Neural Network
计算机科学, 2021, 48(8): 322-327. https://doi.org/10.11896/jsjkx.200600122
[5] 冉孟元, 刘礼, 李艳德, 王珊珊.
基于惯性传感器融合控制算法的聋哑手语识别
Deaf Sign Language Recognition Based on Inertial Sensor Fusion Control Algorithm
计算机科学, 2021, 48(2): 231-237. https://doi.org/10.11896/jsjkx.191200143
[6] 刘肖, 袁冠, 张艳梅, 闫秋艳, 王志晓.
基于自适应多分类器融合的手势识别
Hand Gesture Recognition Based on Self-adaptive Multi-classifiers Fusion
计算机科学, 2020, 47(7): 103-110. https://doi.org/10.11896/jsjkx.200100073
[7] 景雨, 祁瑞华, 刘建鑫, 刘朝霞.
基于改进多尺度深度卷积网络的手势识别算法
Gesture Recognition Algorithm Based on Improved Multiscale Deep Convolutional Neural Network
计算机科学, 2020, 47(6): 180-183. https://doi.org/10.11896/jsjkx.200200030
[8] 程时伟, 陈一健, 徐静如, 张柳新, 吴剑锋, 孙凌云.
一种基于脑电信号的眼动方向分类方法
Approach to Classification of Eye Movement Directions Based on EEG Signal
计算机科学, 2020, 47(4): 112-118. https://doi.org/10.11896/jsjkx.190200342
[9] 程时伟, 齐文杰.
基于动态轨迹的眼动跟踪隐式标定方法
Dynamic Trajectory Based Implicit Calibration Method for Eye Tracking
计算机科学, 2019, 46(8): 282-291. https://doi.org/10.11896/j.issn.1002-137X.2019.08.047
[10] 韩笑, 张晶, 李月龙.
基于手势几何分布特征的手势识别
Gesture Recognition Based on Hand Geometric Distribution Feature
计算机科学, 2019, 46(6A): 246-249.
[11] 李愚, 柴国钟, 卢纯福, 唐智川.
基于增量自适应学习的在线肌电手势识别
On-line sEMG Hand Gesture Recognition Based on Incremental Adaptive Learning
计算机科学, 2019, 46(4): 274-279. https://doi.org/10.11896/j.issn.1002-137X.2019.04.043
[12] 刘佳慧, 王昱洁, 雷艺.
基于LSTM的CSI手势识别方法
CSI Gesture Recognition Method Based on LSTM
计算机科学, 2019, 46(11A): 283-288.
[13] 宋一凡, 张鹏, 刘立波.
基于视觉手势识别的人机交互系统
Human-machine Interaction System with Vision-based Gesture Recognition
计算机科学, 2019, 46(11A): 570-574.
[14] 陈甜甜, 姚璜, 左明章, 田元, 杨梦婷.
基于深度信息的动态手势识别综述
Review of Dynamic Gesture Recognition Based on Depth Information
计算机科学, 2018, 45(12): 42-51. https://doi.org/10.11896/j.issn.1002-137X.2018.12.006
[15] 周婧,陈庙红,吴豪杰.
基于惯性导航的平面航迹推算的研究
Research on Plane Dead Reckoning Based on Inertial Navigation System
计算机科学, 2017, 44(Z6): 582-586. https://doi.org/10.11896/j.issn.1002-137X.2017.6A.131
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!