Computer Science ›› 2024, Vol. 51 ›› Issue (10): 40-49.doi: 10.11896/jsjkx.240400084

• Technology and Application of Intelligent Education • Previous Articles     Next Articles

Survey on Intelligent Analysis Techniques for Classroom Teacher-Student Interaction Research

CUI Jiajun1, KANG Lu1, MA Miao1,2   

  1. 1 School of Computer Science,Shaanxi Normal University,Xi'an 710119,China
    2 Key Laboratory of Modern Teaching Technology of Ministry of Education(Shaanxi Normal University),Xi'an 710062,China
  • Received:2024-04-15 Revised:2024-07-03 Online:2024-10-15 Published:2024-10-11
  • About author:CUI Jiajun,born in 2000,postgraduate.His main research interests include intelligent educational technology and so on.
    MA Miao,born in 1977,Ph.D,professor,Ph.D supervisor.Her main research interests include image processing,video analysis and smart education.
  • Supported by:
    National Natural Science Foundation of China(62377031) and Key Research and Development Program of Shaanxi Province(2023-YBGY-241).

Abstract: With the popularization and continuous development of education informatization,a huge amount of classroom data such as video,image,voice,text are recorded.How to effectively analyze these multimodal data and mine the classroom teacher-student interaction information can not only help teachers find the problems in teaching and adjust the teaching content in time to improve the quality of teaching,moreover,it is an important link to implement the concept of “human-centered” education and promote modern education towards intelligence,personalization and digitalization.The paper firstly discusses the traditional ana-lysis methods of teacher-student interaction behaviors at home and abroad.Then,it classifies and analyzes the current research status of intelligent analysis techniques for classroom teacher-student interaction from different perspectives,such as video,image,voice,text and multimodal.Next,a technical process for classroom teacher-student interaction intelligent analysis is proposed,including core elements,data forms,key technologies,results presentation and application scenarios.Finally,the advantages and disadvantages of the current multimodal intelligent analysis technology for classroom teacher-student interaction are summarized,as well as the challenges and future directions.

Key words: Intelligent educational technology, Classroom teacher-student interaction, Multimodal data, “Human-centered” teaching philosophy

CLC Number: 

  • TP391
[1]LU Y Y,CHEN Z Z,CHEN R,et al.Research on the application framework of intelligent technology to promote teachers'classroom teaching behavior evaluation[J].Modern Educational Technology,2022,32(12):76-84.
[2]EDMUND J,JOHN B.Interaction analysis:theory research and application[J].Behavior Theories,1967(100):402.
[3]MOSKOWITZ G.Interaction analysis-a new modern languagefor supervisors[J].Foreign Language Annals,2010,5(2):211-221.
[4]HOWE C,ABEDIN M.Classroom dialogue:a systematic review across four decades of research[J].Cambridge Journal of Education,2013,43(3):325-356.
[5]CHEN Y S.Multidimensional classroom interaction analysisbased on speech recognition [D].Wuhan:Central China Normal University,2021.
[6]NING H,WU J H.Establishing a connection between quantita-tive structure and meaning understanding-an improved application of Flanders interactive analysis technology[J].Educational Research,2003,24(5):23-27.
[7]JIN J F,GU X Q.Analysis and research on classroom teaching behavior in the information technology environment[J].China Education Technology,2010(9):82-86.
[8]FANG H G,GAO C Z,CHEN J.Improved Flanders interactive analysis system and its application[J].China Education Technology,2012(10):109-113.
[9]LI H M,BIAN P,XU M Q.Development of a dual-coding analysis system for interactive teaching of mathematics classroom in a smart classroom environment and mining of behavioral patterns[J].Modern Educational Technology,2024,34(3):105-115.
[10]SHAN Y J.Using S-T analysis method to analyze classroomteaching problems in educational technology professional courses[J].Modern Educational Technology,2008(10):29-31.
[11]LIU F,LIU Y,HUANG C Y.Comparative analysis of teaching process based on S-T analysis method—taking NetEase video open courses as an example[J].China Education Info,2012(11):58-60.
[12]LIU Y,GOU L.Application of S-T analysis method in analysis of geography classroom teaching model[J].Education of Geography,2013,(6):59-60.
[13]LIU L X,WANG P,HE A N,et al.Research on the application of S-T analysis method in high school chemistry teaching analysis[J].Education in Chemistry,2014(1):27-30.
[14]LI M F,HE F.Analysis of teaching behavior in the “Practice-Feedback” link in the smart classroom-video analysis based on the 19th provincial first-class mathematics lesson example[J].Modern Educational Technology,2019,29(6):62-68.
[15]CHEN J T.Research on intelligent image recognition analysis ofclassroom behavior [D].Hangzhou:Zhejiang University,2019.
[16]XU J H.Research on intelligent recognition of student class-room behavior based on video streaming [D].Fuzhou:Fujian Normal University,2022.
[17]LI M,LIU M,JIANG Z,et al.Multimodal emotion recognition and state analysis of classroom video and audio based on deep neural network[J].Journal of Interconnection Networks,2022,22(Supp4):2146011.
[18]MA L D,CAO Y Z,WU S R.Research on teacher-student interaction in high-quality information-based mathematics classrooms in primary schools based on video case analysis[J].Journal of Beijing Institute of Education,2023,37(5):44-54.
[19]LIU J L.Research on auxiliary tools for segmentation and annotation of video teaching sessions in high school information technology classrooms [D].Wuhan:Central China Normal University,2023.
[20]ZHOU P X,DENG W,GUO P Y,et al.Research on intelligent recognition of S-T behaviors in classroom teaching videos[J].Modern Educational Technology,2018,28(6):54-59.
[21]ZHOU J,RAN F,LI G,et al.Classroom learning status assessment based on deep learning[J].Mathematical Problems in Engineering,2022,2022:1-9.
[22]PABBA C,KUMAR P.An intelligent system for monitoringstudents' engagement in large classroom teaching through facial expression recognition [J].Expert Systems,2022,39(1):e12839.
[23]ZHOU Y H,ZHANG H Y,WANG Y B,et al.Classroom behavior recognitionalgorithm based on AlphaPose[J].Information Technology and Informatization,2023(12):204-208.
[24]ZHANG G,WANG L,CHEN Z.Hand-raising gesture detection in classroom with spatial context augmentation and dilated convolution[J].Computers & Graphics,2023,110:151-161.
[25]TANG J,ZHANG P,ZHANG J.Design and implementation of intelligent evaluation system based on pattern recognition for microteaching skills training[J].International Journal of Innovative Computing,Information and Control,2023,1(19):153-162.
[26]FAN Z J.Design and implementation of classroom teacher-student interactive behavior analysis system [D].Wuhan:Huazhong University of Science and Technology,2018.
[27]YANG N Y.Research on the characteristics and effects ofteachers'classroom emotional work [D].Wuhan:Central China Normal University,2020.
[28]LUO Z Y,ZHAO Q Q,DUAN F Q.Automatic analysis of classroom teaching process based on teachers'near-field speech[J].Modern Educational Technology,2021,31(8):76-84.
[29]ZHENG Z W,HUANG Y T.Research on classroom interaction behavior analysis algorithm based on audio and video[C]//Proceedings of the International Conference on Control and Computer Vision.2022:127-133.
[30]HE Y,GONG Y Q.Improving the quality of online learning:a study on teacher-student interaction based on network multi-modal data analysis[C]//Proceedings of the International Conference on Big Data and Education.2022:311-318.
[31]HAN H,WANG D Q,CAO C.Analysis and research on classroom teaching interaction behavior in 1∶1 digital environment[J].E-Education Research,2015,36(5):89-95.
[32]HUANG Y.Analysis of teachers' teaching speech behavior sup-ported by online course data mining [D].Wuhan:Central China Normal University,2020.
[33]CHEN M,LI M Y.Research on classroom teacher-student verbal interaction behavior based on IRF discourse structure[J].Educational Information Technology,2022,(Z2):3-8,30.
[34]SGIN J,BALYAN R,BANAWAN M P,et al.Pedagogical discourse markers in online algebra learning:unraveling instructor'scommunication using natural language processing[J].Computers &Education,2023,205:104897.
[35]ZHAO Y,KONG X,ZHENG W,et al.Emotion generationmethod in online physical education teaching based on data mi-ning of teacher-student interactions[J].PeerJ Computer Science,2024,10:e1814.
[36]SUSANNE K.Contradictory xplorative Assessment.Multimodal Teacher/Student Interaction in Scandinavian Digital Lear-ning Environments[J].Journal of Education and Training Stu-dies,2018,6(2):133-148.
[37]WU L,CAO Y,DU Q,et al.The analysis path of classroom teacher behavior supported by artificial intelligence[J].Artificial Intelligence in Education and Teaching Assessment,2021,11:235-245.
[38]YU B,LIU Y,CHAN K C C.Multimodal fusion via teacher-student network for indoor action recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2021,35(4):3199-3207.
[39]TONG H,YANG Y J.Research on smart classroom teaching in-teraction based on multi-modal data[J].E-Education Research,2022,43(3):60-68.
[40]WANG M,LUO L,CHEN Z,et al.Intelligent multimodal analysisframework for teacher-student interaction[C]//2022 International Conference on Intelligent Education and Intelligent Research(IEIR).IEEE,2022:65-70.
[41]LI X J,LIU Q T,WU L J,et al.Dynamic collaborative analysis of teachers and students' multimodal interactive behaviors in hybrid synchronous classrooms[J].E-Education Research,2022,43(8):43-50.
[42]DUBOVI I.Cognitive and emotional engagement while learning with VR:the perspective of multimodal methodology[J].Computers & Education,2022,183:104495.
[43]VIVANTE I,VEDDER-WEISS D.Examining science teachers'engagement in professional development:a multimodal situated perspective[J].Journal of Research in Science Teaching,2023,60(7):1401-1430.
[44]XU Y,ZHANG J,ZHANG Q,et al.ViTPose:simple visiontransformer baselines for human pose estimation[J].Advances in Neural Information Processing Systems,2022,35:38571-38584.
[45]ZHENG C,WU W,CHEN C,et al.Deep learning-based human pose estimation:a survey[J].ACM Computing Surveys,2023,56(1):1-37.
[46]LI J N,WANG D K,ZHANG S L.Deep-learning-based 2D human pose estimation:present and future[J].Chinese Journal of Computers,2024,47(1):231-250.
[47]SHI D,ZHONG Y,CAO Q,et al.Tridet:temporal action detection with relative boundary modeling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:18857-18866.
[48]HU K,SHEN C,WANG T,et al.Overview of temporal actiondetection based on deep learning[J].Artificial Intelligence Review,2024,57(2):26.
[49]HUANG X K,ZHANG J Y,WANG X Y,et al.Survey of dense video captioning[J].Computer Engineering and Applications,2023,59(12):28-48.
[50]TANG P J,WANG H L.From video to language:survey of vi-deo captioning and description[J].Journal of Automation,2022,48(2):375-397.
[51]SHEN X,LI D,ZHOU J,et al.Fine-grained audible video description[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:10585-10596.
[52]XU H,ZHANG K,TIAN Y J,et al.Review of deep neural network-Based image Caption[J].Computer Engineering and Applications,2021,57(9):14.
[53]SHI Y L,YANG W Z,DU H X,et al.A review of deep learning-based image description[J].Chinese Journal of Electronics,2021,49(10):2048-2060.
[54]RAHMANI H,BENNAMOUN M,KE Q,et al.Human action recognition from various data modalities:a review[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,45(3):3200-3225.
[55]BI C Y,LIU Y.A survey of video human action recognitionbased on deep learning[J].Journal of Graphics,2023,44(4):625-639.
[56]LIN L,ZHANG J,LIU J.Actionlet-dependent contrastive lear-ning for unsupervised skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:2363-2372.
[57]WU T,CAO M,GAO Z,et al.Stmixer:a one-stage sparse action detector[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:14720-14729.
[58]XIE Y G,WANG Q.Summary of dynamic gesture recognition based on vision[J].Computer Engineering and Applications,2021,57(22):68-77.
[59]WANG R P,WU S H,ZHANG M H,et al.Review of vision-based neural network 3D dynamic gesture recognition methods[J].Computer Science,2024,51(4):193-208.
[60]BAO H,HU W X.Technical features of facial expression recognition and analysis of educational application scenarios[J].Electronic Engineering & Product World,2023,30(3):39-43.
[61]LIN S N,ZHAO R,ZHANG W.A review of deep learning-based research on student expression recognition[J].Information Technology & Informatization,2023(11):188-194.
[62]MA H,TANG B R,ZHANG Y,et al.Survey on speech recognition[J].Computer Systems & Applications,2022,31(1):1-10.
[63]WANG A H,ZHANG L,SONG W Y,et al.Review of end-to-end streaming speech recognition[J].Computer Engineering and Applications,2023,59(2):22-33.
[64]WANG W C.Research on speaker clustering and identification methods based on deep convolutional network [D].Guangzhou:South China University of Technology,2021.
[65]BECCARO W,RAMÍREZ M A,LIAW W, et al.Analysis oforal exams with speaker diarization and speech emotion recognition:a case study[J].IEEE Transactions on Education,2023,67(1):74-86.
[66]LUO Y Y,QIN X D,XIE Y P,et al.Intelligent data visualization analysis techniques:a survey[J].Journal of Software,2024,35(1):356-404.
[1] ZHANG Lianfu, TAN Zuowen. Federated Learning Privacy-preserving Approach for Multimodal Medical Data [J]. Computer Science, 2023, 50(11A): 230800021-8.
[2] ZHAO Liang, ZHANG Jie, CHEN Zhi-kui. Adaptive Multimodal Robust Feature Learning Based on Dual Graph-regularization [J]. Computer Science, 2022, 49(4): 124-133.
[3] JIAN Song-lei, LU Kai. Survey on Representation Learning of Complex Heterogeneous Data [J]. Computer Science, 2020, 47(2): 1-9.
[4] LIN Min-hong, MENG Zu-qiang. Multimodal Sentiment Analysis Based on Attention Neural Network [J]. Computer Science, 2020, 47(11A): 508-514.
[5] XU Cheng-hao, GUO Bin, OUYANG Yi, ZHAI Shu-ying and YU Zhi-wen. Event Sensing and Multimodal Event Vein Generation Leveraging Social Media [J]. Computer Science, 2017, 44(Z6): 33-36.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!