Computer Science

Computer Science ›› 2022, Vol. 49 ›› Issue (11A): 211000049-6.doi: 10.11896/jsjkx.211000049

• Artificial Intelligence • Previous Articles     Next Articles

Online Learning Emotion Recognition Based on Videos

WEI Yan-tao1,2, LUO Jie-lin1,2, HU Mei-jia1,2, LI Wen-hao1, YAO Huang1   

  1. 1 Faculty of Artificial Intelligence in Education,Central China Normal University,Wuhan 430079,China
    2 Hubei Research Center for Educational Informationization,Central China Normal University,Wuhan 430079,China
  • Online:2022-11-10 Published:2022-11-21
  • About author:WEI Yan-tao,born in 1983,Ph.D,associate professor,is a member of China Computer Federation.His main research interests include educational artificial intelligence,computer vision and machine learning.
    HU Mei-jia,born in 1997,postgraduate.Her main research interests include educational artificial intelligence,computer vision and machine learning.
  • Supported by:
    Humanities and Social Sciences of China MOE(20YJC880100).

PDF (PC)

429

Abstract: With the normalization of epidemic prevention and control,online learning has become one of the main forms of daily teaching activities.However,with the large-scale development of online learning activities,the problem of “emotional loss” is increasingly prominent,which has become the main reason for the low completion rate of online learning.Aiming to deal with the above problems,the non-invasive online learning emotion state recognition method using video data is discussed.Firstly,the facial videos and heart rate data of 22 students learning online are collected to construct a bimodal online learning emotion database.Secondly,the frame attention network is used to extract facial expression features from the learning video and recognize the emotional state of online learning,and its recognition accuracy reaches 87.8%.Finally,the application of the video heart rate recognition method in online learning emotion analysis is discussed.Research results show that the heart rate level in the confused state is significant.Starting from learning video data mining,focusing on learning emotion recognition based on facial expressions and video heart rate,which provides a new idea for improving the perception of emotional state in online learning.

Key words: Learning emotion, Online Learning, Facial expression, Heart rate, Emotion recognition

CLC Number: 

  • G40-057
[1]HUANG C Q,TU Y X,YU J H,et al.Research and implemen-tation of Data-driven Online Learning Burnout Early Warning Model[J].E-education Research,2021,42(2):47-54.
[2]GUO C L.Solutions to the Lack of Teaching Emotion between Teachers and Students in Online Teaching under the Epidemic Situation[J].The Chinese Journal of ICT in Education,2021(7):65-69.
[3]HUANG C.Research on Learner Emotion Modeling and Application for E-learning[D].Wuhan:Central China Normal University,2014.
[4]WEI R J,DING Y Z,ZHANG L,et al.Design and Implementation of Emotion Recognition Module in Online Learning System[J].Modern Educational Technology,2014(3):115-122.
[5]CAO X M,ZHANG Y H,PAN M,et al.Research on Recognition Method of Learning Participation from the Perspective of Artificial Intelligence-Analysis of Deep Learning Experiment based on Multimodal Data Fusion[J].Journal of Distance Education,2019(1):32-44.
[6]TAO X M,CHEN X Y.Research on emotion recognition based on eye movement features in online learning environment[J].Computer Technology and Development,2021,31(3):186-190.
[7]YU Z T,LI X B,ZHAO G Y.Emotion Recognition and Education[J].artificial intelligence,2019(3):29-36.
[8]PISE A,VADAPALL H,SANDERS I.Facial emotion recognition using temporal relational network:an application to E-learning[J].Multimedia Tools and Applications,2020(2):1-12.
[9]DINDAR M,JARVELA S,AHOLA S,et al.Leaders and fol-lowers identified by emotional mimicry during collaborative learning:A facial expression recognition study on emotional valence[J].IEEE Transactions on Affective Computing,2020,13(3):1390-1400.
[10]PEKRUN R,VOGL E,MUIS K R,et al.Measuring emotions during epistemic activities:the Epistemically-Related Emotion Scales[J].Cognition & emotion,2017,31(6):1268-1276.
[11]ASHWIN T S,RAM M R G.Automatic detection of students’ affective states in classroom environment using hybrid convolutional neural networks[J].Education and information technologies,2020,25(2):1387-1415.
[12]XU Z G.Emotion Recognition of Learning Picture in Intelligent Learning Environment and Its Application[D].Jinan:Shandong Normal University,2019.
[13]MA Y T.Design and Implementation of Emotion Analysis Mo-del for Online Learning based on Multimodality [D].Nanjing:Nanjing Normal University,2019.
[14]JIA L Y,ZHANG Z H,ZHAO X Y,et al.Classroom Student State Analysis Based on Artificial Intelligence Video Processing [J].Modern Educational Technology,2019,29(12):82-88.
[15]CHEN X,CHENG J,SONG R,et al.Video-Based Heart Rate Measurement:Recent Advances and Future Prospects[J].IEEE Transactions on Instrumentation and Measurement,2018,68(10):3600-3615.
[16]LUCEY P,COHN J F,KANADE T,et al.The Extended Cohn Kanade Dataset(CK+):A complete dataset for action unit and emotion-specified expression[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.2010:94-101.
[17]BARROS P,CHURAMAN N,LAKOMKIN E,et al.The omg-emotionbehavior dataset[C]//International Joint Conference on Neural Networks (IJCNN).2018:1-17.
[18]LI J,CHEN Y,XIAO S,et al.Estimation of affective level in the wild with multiple memory networks[C]//IEEE Conference on Computer Vision and Pattern Recognition Workshops.2017:1-8.
[19]WIEM M B H,LACHIRI Z.Emotion classification in arousalvalence model using MAHNOB-HCI database[J].International Journal of Advanced Computer Science and Applications,2017,8(3):318-323.
[20]XY Z G,ZHANG G W,MENG X Z,et al.Learners’ Emotion Recognition and Application Based on Deep Learning[J].e-EDUCATION RESEARCH,2019,40(2):87-94.
[21]HE X K,GAO Q,LI Y Y,et al.Research on SpontaneousLearning Expression Recognition Method Based on Deep Learning Model [J].Computer Applications and Software,2019,36(3):180-186.
[22]BAKER R S,D’MELLO S K,RODRIGO M,et al.Better to be frustrated than bored:The incidence and prevalence of affect during interactions with three computer-based learning environments [J].International Journal of Human-Computer Studies,2010,68(4):223-241.
[23]GUO Y,ZHANG L,HU Y,et al.Ms-celeb-1m:A dataset and benchmark for large-scale face recognition[C]//European Conference on Computer Vision.Cham:Springer,2016:87-102.
[24]BARSOUM E,ZHANG C,FERRER C C,et al.Training deep networks for facial expression recognition with crowd-sourced label distribution [C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction.2016:279-283.
[25]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[26]MENG D,PENG X,WANG K,et al.Frame attention networks for facial expression recognition in videos[C]//2019 IEEE International Conference on Image Processing(ICIP).IEEE,2019:3866-3870.
[27]CHEN X,CHENG J,SONG R,et al.Video-based heart ratemeasurement:Recent advances and future prospects[J].IEEE Transactions on Instrumentation and Measurement,2018,68(10):3600-3615.
[28]NIU X S,HANG H,SHAN S G.Review of Physiological Index Measurement Methods Based on R-ppg[J].Journal of Image andGraphics,2020,25(11):2321-2336.
[29]WANG W,DENG B A C,STUIJK S,et al.Algorithmic principles of remote PPG[J].IEEE Transactions on Biomedical Engineering,2016,64(7):1479-1491.
[1] XU Ming-ke, ZHANG Fan. Head Fusion:A Method to Improve Accuracy and Robustness of Speech Emotion Recognition [J]. Computer Science, 2022, 49(7): 132-141.
[2] GAO Yue, FU Xiang-ling, OUYANG Tian-xiong, CHEN Song-ling, YAN Chen-wei. EEG Emotion Recognition Based on Spatiotemporal Self-Adaptive Graph ConvolutionalNeural Network [J]. Computer Science, 2022, 49(4): 30-36.
[3] ZENG Zi-lin, HU Zhi-gang, SHANG Peng, WANG Xin-zheng, FU Dong-liao. Non-contact Heart Rate Detection Based on Facial Video [J]. Computer Science, 2022, 49(11A): 211000182-6.
[4] QIAN Tian-tian, ZHANG Fan. Emotion Recognition System Based on Distributed Edge Computing [J]. Computer Science, 2021, 48(6A): 638-643.
[5] CHAI Bing, LI Dong-dong, WANG Zhe, GAO Da-qi. EEG Emotion Recognition Based on Frequency and Channel Convolutional Attention [J]. Computer Science, 2021, 48(12): 312-318.
[6] YE Ya-nan, CHI Jing, YU Zhi-ping, ZHAN Yu-liand ZHANG Cai-ming. Expression Animation Synthesis Based on Improved CycleGan Model and Region Segmentation [J]. Computer Science, 2020, 47(9): 142-149.
[7] LIU Ling-yun, QIAN Hui, XING Hong-jie, DONG Chun-ru, ZHANG Feng. Incremental Classification Model Based on Q-learning Algorithm [J]. Computer Science, 2020, 47(8): 171-177.
[8] XIAO Xiao and KONG Fan-zhi. New Representation of Facial Affect Based on Triangular Coordinate System [J]. Computer Science, 2020, 47(6A): 250-253.
[9] KONG Fang, LI Qi-zhi, LI Shuai. Survey on Online Influence Maximization [J]. Computer Science, 2020, 47(5): 7-13.
[10] ZHENG Chun-jun, WANG Chun-li, JIA Ning. Survey of Acoustic Feature Extraction in Speech Tasks [J]. Computer Science, 2020, 47(5): 110-119.
[11] WAN Jia-shan, CHEN Lei, WU Jin-hua, GAO Chao. Persona Based Social User Modeling Using KD-Tree [J]. Computer Science, 2019, 46(6A): 442-445.
[12] HE Xiao-wen, HU Yi-fei, WANG Hai-ping, CHEN Mo. Online Learning Nonnegative Matrix Factorization [J]. Computer Science, 2019, 46(6A): 473-477.
[13] JIA Ning, LI Ying-da. Construction of Personalized Health Monitoring Platform Based on Intelligent Wearable Device [J]. Computer Science, 2019, 46(6A): 566-570.
[14] LI De-quan, DONG Qiao, ZHOU Yue-jin. Distributed Online Conditional Gradient Optimization Algorithm [J]. Computer Science, 2019, 46(3): 332-337.
[15] JIANG Bin,GAN Yong,ZHANG Huan-long,ZHANG Qiu-wen. Survey on Non-frontal Facial Expression Recognition Methods [J]. Computer Science, 2019, 46(3): 53-62.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.