Computer Science

Computer Science ›› 2023, Vol. 50 ›› Issue (9): 184-191.doi: 10.11896/jsjkx.221100043

• Database & Big Data & Data Science • Previous Articles     Next Articles

Sign Language Animation Splicing Model Based on LpTransformer Network

HUANG Hanqiang1,2, XING Yunbing2,3, SHEN Jianfei2,3, FAN Feiyi2   

  1. 1 Henan Institute of Advanced Technology,Zhengzhou University,Zhengzhou 450000,China
    2 Institute of Computing Technology,Chinese Academy of Sciences,Beijing 100000,China
    3 Shandong Industrial Technology Research Institute Intelligent Computing Research Institute,Jinan 250000,China
  • Received:2022-11-07 Revised:2023-02-28 Online:2023-09-15 Published:2023-09-01
  • About author:HUANG Hanqiang,born in 1998,postgraduate.His main research interests include graphic image processing and sign language processing.
    XING Yunbing,born in 1982,master,senior engineer.His main research interests include sign language and human-computer interaction.
  • Supported by:
    National Key Research and Development Program of China(2018YFC2002603).

PDF (PC)

1069

Abstract: Sign language animation splicing is a hot topic.With the continuous development of machine learning technology,especially the gradual maturity of deep learning related technologies,the speed and quality of sign language animation splicing are constantly improving.When splicing sign language words into sentences,the corresponding animation also needs to be spliced.Traditional algorithms use distance loss to find the best splicing position when splicing animation,and use linear or spherical interpolation to generate transition frames.This splicing algorithm not only has obvious defects in efficiency and flexibility,but also gene-rates unnatural sign language animation.In order to solve the above problems,LpTransformer model is proposed to predict the splicing position and generate transition frames.Experiment results show that the prediction accuracy of LpTransformer's transition frames reaches 99%,which is superior to ConvS2S,LSTM and Transformer,and its splicing speed is five times faster than Transformer,so it can achieve real-time splicing.

Key words: Sign language animation splicing, Deep learning, LpTransformer, Splicing position, Transition frames

CLC Number: 

  • TP183
[1]ZHU T T.The research of chinese sign language video synthesis aided by 3D information [D].Beijing:Beijing University of Technology,2014.
[2]CHEN J X.Study on key technologies of the chinese sign language synthesis based on the video stitching [D].Hefei:University of Science and Technology of China,2017.
[3]ZHAO H N.Chinese sign language news broadcasting system based on virtual human technology [D].Harbin:Harbin Institute of Technology,2008.
[4]DUARTE A C.Cross-modal neural sign language translation[C]//Proceedings of the 27th ACM International Conference on Multimedia.Nice:ACM,2019:1650-1654.
[5]KAPOOR P,MUKHOPADHYAY R,HEGDE S B,et al.To-wards Automatic Speech to Sign Language Generation[C]//Interspeech 2021,22nd Annual Conference of the International Speech Communication Association.Brno:ISCA,2021:3700-3704.
[6]XIAO Q,QIN M,YIN Y.Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing people[J].Neural networks,2020,125:41-55.
[7]SAUNDERS B,CAMGOZ N C,BOWDEN R.Progressive transformers for end-to-end sign language production[C]//European Conference on Computer Vision.Glasgow:Springer,2020:687-705.
[8]ZELINKA J,KANIS J.Neural sign language synthesis:Words are our glosses[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.Snowmass Village:IEEE,2020:3395-3403.
[9]SUNDERS B,CAMGOZ N C,BOWDEN R.Continuous 3dmulti-channel sign language production via progressive transformers and mixture density networks[J].International Journal of Computer Vision,2021,129(7):2113-2135.
[10]HUANG W,PAN W,ZHAO Z,et al.Towards Fast and High-Quality Sign Language Production[C]//Proceedings of the 29th ACM International Conference on Multimedia.China:ACM,2021:3172-3181.
[11]ZHOU C,LAI Z,WANG S,et al.Learning a deep motion interpolation network for human skeleton animations[J].Computer Animation and Virtual Worlds,2021,32(3/4):e2003.
[12]SAUNDERS B,CAMGOZ N C,BOWDEN R.Skeletal Graph Self-Attention:Embedding a Skeleton Inductive Bias into Sign Language Production[J].arXiv:2112.05277,2021.
[13]ZHANG Z,XUE W,HUANG W,et al.Effective Video Frame Acquisition for Image Stitching[J].IEEE access,2020,8:217086-217097.
[14]LIU Q,SU X,ZHANG L,et al.Panoramic video stitching ofdual cameras based on spatio-temporal seam optimization[J].Multimedia Tools and Applications,2020,79(5):3107-3124.
[15]VASUHI S,SAMYDURAI A,VIJAYAKUMAR M.Multica-mera Video Stitching for Multiple Human Tracking[J].International Journalof Computer Vision and Image Processing (IJCVIP),2021,11(1):17-38.
[16]CAO W.Applying image registration algorithm combined withCNN model to video image stitching[J].The Journal of Supercomputing,2021,77(12):13879-13896.
[17]DAS A,RAUN E S K,KJARGAARD M B.Cam-stitch:Trajectory cavity stitching method for stereo vision cameras in a public building[C]//Proceedings of the First International Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things.New York:Association for Computing Machinery,2019:8-14.
[18]GEHRING J,AULI M,GRANGIE D,et al.Convolutional sequence to sequence learning[C]//International Conference on Machine Learning.Sydney:PMLR,2017:1243-1252.
[19]VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[J].arXiv:1706.03762,2017.
[1] ZHAO Mingmin, YANG Qiuhui, HONG Mei, CAI Chuang. Smart Contract Fuzzing Based on Deep Learning and Information Feedback [J]. Computer Science, 2023, 50(9): 117-122.
[2] LI Haiming, ZHU Zhiheng, LIU Lei, GUO Chenkai. Multi-task Graph-embedding Deep Prediction Model for Mobile App Rating Recommendation [J]. Computer Science, 2023, 50(9): 160-167.
[3] ZHU Ye, HAO Yingguang, WANG Hongyu. Deep Learning Based Salient Object Detection in Infrared Video [J]. Computer Science, 2023, 50(9): 227-234.
[4] ZHANG Yian, YANG Ying, REN Gang, WANG Gang. Study on Multimodal Online Reviews Helpfulness Prediction Based on Attention Mechanism [J]. Computer Science, 2023, 50(8): 37-44.
[5] SONG Xinyang, YAN Zhiyuan, SUN Muyi, DAI Linlin, LI Qi, SUN Zhenan. Review of Talking Face Generation [J]. Computer Science, 2023, 50(8): 68-78.
[6] WANG Xu, WU Yanxia, ZHANG Xue, HONG Ruize, LI Guangsheng. Survey of Rotating Object Detection Research in Computer Vision [J]. Computer Science, 2023, 50(8): 79-92.
[7] ZHOU Ziyi, XIONG Hailing. Image Captioning Optimization Strategy Based on Deep Learning [J]. Computer Science, 2023, 50(8): 99-110.
[8] ZHANG Xiao, DONG Hongbin. Lightweight Multi-view Stereo Integrating Coarse Cost Volume and Bilateral Grid [J]. Computer Science, 2023, 50(8): 125-132.
[9] WANG Yu, WANG Zuchao, PAN Rui. Survey of DGA Domain Name Detection Based on Character Feature [J]. Computer Science, 2023, 50(8): 251-259.
[10] LI Kun, GUO Wei, ZHANG Fan, DU Jiayu, YANG Meiyue. Adversarial Malware Generation Method Based on Genetic Algorithm [J]. Computer Science, 2023, 50(7): 325-331.
[11] WANG Mingxia, XIONG Yun. Disease Diagnosis Prediction Algorithm Based on Contrastive Learning [J]. Computer Science, 2023, 50(7): 46-52.
[12] SHEN Zhehui, WANG Kailai, KONG Xiangjie. Exploring Station Spatio-Temporal Mobility Pattern:A Short and Long-term Traffic Prediction Framework [J]. Computer Science, 2023, 50(7): 98-106.
[13] HUO Weile, JING Tao, REN Shuang. Review of 3D Object Detection for Autonomous Driving [J]. Computer Science, 2023, 50(7): 107-118.
[14] ZHOU Bo, JIANG Peifeng, DUAN Chang, LUO Yuetong. Study on Single Background Object Detection Oriented Improved-RetinaNet Model and Its Application [J]. Computer Science, 2023, 50(7): 137-142.
[15] MAO Huihui, ZHAO Xiaole, DU Shengdong, TENG Fei, LI Tianrui. Short-term Subway Passenger Flow Forecasting Based on Graphical Embedding of Temporal Knowledge [J]. Computer Science, 2023, 50(7): 213-220.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.