计算机科学 ›› 2024, Vol. 51 ›› Issue (10): 40-49.doi: 10.11896/jsjkx.240400084
崔家郡1, 康璐1, 马苗1,2
CUI Jiajun1, KANG Lu1, MA Miao1,2
摘要： 随着教育信息化的普及与不断发展,视频、图像、语音、文本等海量课堂数据被记录下来。对这些多模态数据进行有效分析,挖掘课堂师生交互信息,不仅能够帮助教师及时发现教学中存在的问题,及时调整教学内容以提高教学质量,而且是落实“以人为本”教学理念,推进现代教育走向智能化、个性化和数字化的重要手段。文中首先论述国内外师生交互行为的传统分析方法;然后从视频、图像、语音、文本及多模态等不同角度分类梳理课堂师生交互智能分析技术的研究现状,归纳总结课堂师生交互智能分析的核心要素、数据形式、关键技术、结果呈现和应用场景;最后分析课堂师生交互的多模态智能分析技术的优势与不足以及面临的挑战和未来趋势。
中图分类号:
| [1]LU Y Y,CHEN Z Z,CHEN R,et al.Research on the application framework of intelligent technology to promote teachers'classroom teaching behavior evaluation[J].Modern Educational Technology,2022,32(12):76-84. [2]EDMUND J,JOHN B.Interaction analysis:theory research and application[J].Behavior Theories,1967(100):402. [3]MOSKOWITZ G.Interaction analysis－a new modern languagefor supervisors[J].Foreign Language Annals,2010,5(2):211-221. [4]HOWE C,ABEDIN M.Classroom dialogue:a systematic review across four decades of research[J].Cambridge Journal of Education,2013,43(3):325-356. [5]CHEN Y S.Multidimensional classroom interaction analysisbased on speech recognition [D].Wuhan:Central China Normal University,2021. [6]NING H,WU J H.Establishing a connection between quantita-tive structure and meaning understanding－an improved application of Flanders interactive analysis technology[J].Educational Research,2003,24(5):23-27. [7]JIN J F,GU X Q.Analysis and research on classroom teaching behavior in the information technology environment[J].China Education Technology,2010(9):82-86. [8]FANG H G,GAO C Z,CHEN J.Improved Flanders interactive analysis system and its application[J].China Education Technology,2012(10):109-113. [9]LI H M,BIAN P,XU M Q.Development of a dual-coding analysis system for interactive teaching of mathematics classroom in a smart classroom environment and mining of behavioral patterns[J].Modern Educational Technology,2024,34(3):105-115. [10]SHAN Y J.Using S-T analysis method to analyze classroomteaching problems in educational technology professional courses[J].Modern Educational Technology,2008(10):29-31. [11]LIU F,LIU Y,HUANG C Y.Comparative analysis of teaching process based on S-T analysis method—taking NetEase video open courses as an example[J].China Education Info,2012(11):58-60. [12]LIU Y,GOU L.Application of S-T analysis method in analysis of geography classroom teaching model[J].Education of Geography,2013,(6):59-60. [13]LIU L X,WANG P,HE A N,et al.Research on the application of S-T analysis method in high school chemistry teaching analysis[J].Education in Chemistry,2014(1):27-30. [14]LI M F,HE F.Analysis of teaching behavior in the “Practice-Feedback” link in the smart classroom-video analysis based on the 19th provincial first-class mathematics lesson example[J].Modern Educational Technology,2019,29(6):62-68. [15]CHEN J T.Research on intelligent image recognition analysis ofclassroom behavior [D].Hangzhou:Zhejiang University,2019. [16]XU J H.Research on intelligent recognition of student class-room behavior based on video streaming [D].Fuzhou:Fujian Normal University,2022. [17]LI M,LIU M,JIANG Z,et al.Multimodal emotion recognition and state analysis of classroom video and audio based on deep neural network[J].Journal of Interconnection Networks,2022,22(Supp4):2146011. [18]MA L D,CAO Y Z,WU S R.Research on teacher-student interaction in high-quality information-based mathematics classrooms in primary schools based on video case analysis[J].Journal of Beijing Institute of Education,2023,37(5):44-54. [19]LIU J L.Research on auxiliary tools for segmentation and annotation of video teaching sessions in high school information technology classrooms [D].Wuhan:Central China Normal University,2023. [20]ZHOU P X,DENG W,GUO P Y,et al.Research on intelligent recognition of S-T behaviors in classroom teaching videos[J].Modern Educational Technology,2018,28(6):54-59. [21]ZHOU J,RAN F,LI G,et al.Classroom learning status assessment based on deep learning[J].Mathematical Problems in Engineering,2022,2022:1-9. [22]PABBA C,KUMAR P.An intelligent system for monitoringstudents' engagement in large classroom teaching through facial expression recognition [J].Expert Systems,2022,39(1):e12839. [23]ZHOU Y H,ZHANG H Y,WANG Y B,et al.Classroom behavior recognitionalgorithm based on AlphaPose[J].Information Technology and Informatization,2023(12):204-208. [24]ZHANG G,WANG L,CHEN Z.Hand-raising gesture detection in classroom with spatial context augmentation and dilated convolution[J].Computers & Graphics,2023,110:151-161. [25]TANG J,ZHANG P,ZHANG J.Design and implementation of intelligent evaluation system based on pattern recognition for microteaching skills training[J].International Journal of Innovative Computing,Information and Control,2023,1(19):153-162. [26]FAN Z J.Design and implementation of classroom teacher-student interactive behavior analysis system [D].Wuhan:Huazhong University of Science and Technology,2018. [27]YANG N Y.Research on the characteristics and effects ofteachers'classroom emotional work [D].Wuhan:Central China Normal University,2020. [28]LUO Z Y,ZHAO Q Q,DUAN F Q.Automatic analysis of classroom teaching process based on teachers'near-field speech[J].Modern Educational Technology,2021,31(8):76-84. [29]ZHENG Z W,HUANG Y T.Research on classroom interaction behavior analysis algorithm based on audio and video[C]//Proceedings of the International Conference on Control and Computer Vision.2022:127-133. [30]HE Y,GONG Y Q.Improving the quality of online learning:a study on teacher-student interaction based on network multi-modal data analysis[C]//Proceedings of the International Conference on Big Data and Education.2022:311-318. [31]HAN H,WANG D Q,CAO C.Analysis and research on classroom teaching interaction behavior in 1∶1 digital environment[J].E-Education Research,2015,36(5):89-95. [32]HUANG Y.Analysis of teachers' teaching speech behavior sup-ported by online course data mining [D].Wuhan:Central China Normal University,2020. [33]CHEN M,LI M Y.Research on classroom teacher-student verbal interaction behavior based on IRF discourse structure[J].Educational Information Technology,2022,(Z2):3-8,30. [34]SGIN J,BALYAN R,BANAWAN M P,et al.Pedagogical discourse markers in online algebra learning:unraveling instructor'scommunication using natural language processing[J].Computers &Education,2023,205:104897. [35]ZHAO Y,KONG X,ZHENG W,et al.Emotion generationmethod in online physical education teaching based on data mi-ning of teacher-student interactions[J].PeerJ Computer Science,2024,10:e1814. [36]SUSANNE K.Contradictory xplorative Assessment.Multimodal Teacher/Student Interaction in Scandinavian Digital Lear-ning Environments[J].Journal of Education and Training Stu-dies,2018,6(2):133-148. [37]WU L,CAO Y,DU Q,et al.The analysis path of classroom teacher behavior supported by artificial intelligence[J].Artificial Intelligence in Education and Teaching Assessment,2021,11:235-245. [38]YU B,LIU Y,CHAN K C C.Multimodal fusion via teacher-student network for indoor action recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2021,35(4):3199-3207. [39]TONG H,YANG Y J.Research on smart classroom teaching in-teraction based on multi-modal data[J].E-Education Research,2022,43(3):60-68. [40]WANG M,LUO L,CHEN Z,et al.Intelligent multimodal analysisframework for teacher-student interaction[C]//2022 International Conference on Intelligent Education and Intelligent Research(IEIR).IEEE,2022:65-70. [41]LI X J,LIU Q T,WU L J,et al.Dynamic collaborative analysis of teachers and students' multimodal interactive behaviors in hybrid synchronous classrooms[J].E-Education Research,2022,43(8):43-50. [42]DUBOVI I.Cognitive and emotional engagement while learning with VR:the perspective of multimodal methodology[J].Computers & Education,2022,183:104495. [43]VIVANTE I,VEDDER-WEISS D.Examining science teachers'engagement in professional development:a multimodal situated perspective[J].Journal of Research in Science Teaching,2023,60(7):1401-1430. [44]XU Y,ZHANG J,ZHANG Q,et al.ViTPose:simple visiontransformer baselines for human pose estimation[J].Advances in Neural Information Processing Systems,2022,35:38571-38584. [45]ZHENG C,WU W,CHEN C,et al.Deep learning-based human pose estimation:a survey[J].ACM Computing Surveys,2023,56(1):1-37. [46]LI J N,WANG D K,ZHANG S L.Deep-learning-based 2D human pose estimation:present and future[J].Chinese Journal of Computers,2024,47(1):231-250. [47]SHI D,ZHONG Y,CAO Q,et al.Tridet:temporal action detection with relative boundary modeling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:18857-18866. [48]HU K,SHEN C,WANG T,et al.Overview of temporal actiondetection based on deep learning[J].Artificial Intelligence Review,2024,57(2):26. [49]HUANG X K,ZHANG J Y,WANG X Y,et al.Survey of dense video captioning[J].Computer Engineering and Applications,2023,59(12):28-48. [50]TANG P J,WANG H L.From video to language:survey of vi-deo captioning and description[J].Journal of Automation,2022,48(2):375-397. [51]SHEN X,LI D,ZHOU J,et al.Fine-grained audible video description[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:10585-10596. [52]XU H,ZHANG K,TIAN Y J,et al.Review of deep neural network-Based image Caption[J].Computer Engineering and Applications,2021,57(9):14. [53]SHI Y L,YANG W Z,DU H X,et al.A review of deep learning-based image description[J].Chinese Journal of Electronics,2021,49(10):2048-2060. [54]RAHMANI H,BENNAMOUN M,KE Q,et al.Human action recognition from various data modalities:a review[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,45(3):3200-3225. [55]BI C Y,LIU Y.A survey of video human action recognitionbased on deep learning[J].Journal of Graphics,2023,44(4):625-639. [56]LIN L,ZHANG J,LIU J.Actionlet-dependent contrastive lear-ning for unsupervised skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:2363-2372. [57]WU T,CAO M,GAO Z,et al.Stmixer:a one-stage sparse action detector[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:14720-14729. [58]XIE Y G,WANG Q.Summary of dynamic gesture recognition based on vision[J].Computer Engineering and Applications,2021,57(22):68-77. [59]WANG R P,WU S H,ZHANG M H,et al.Review of vision-based neural network 3D dynamic gesture recognition methods[J].Computer Science,2024,51(4):193-208. [60]BAO H,HU W X.Technical features of facial expression recognition and analysis of educational application scenarios[J].Electronic Engineering & Product World,2023,30(3):39-43. [61]LIN S N,ZHAO R,ZHANG W.A review of deep learning-based research on student expression recognition[J].Information Technology & Informatization,2023(11):188-194. [62]MA H,TANG B R,ZHANG Y,et al.Survey on speech recognition[J].Computer Systems & Applications,2022,31(1):1-10. [63]WANG A H,ZHANG L,SONG W Y,et al.Review of end-to-end streaming speech recognition[J].Computer Engineering and Applications,2023,59(2):22-33. [64]WANG W C.Research on speaker clustering and identification methods based on deep convolutional network [D].Guangzhou:South China University of Technology,2021. [65]BECCARO W,RAMÍREZ M A,LIAW W, et al.Analysis oforal exams with speaker diarization and speech emotion recognition:a case study[J].IEEE Transactions on Education,2023,67(1):74-86. [66]LUO Y Y,QIN X D,XIE Y P,et al.Intelligent data visualization analysis techniques:a survey[J].Journal of Software,2024,35(1):356-404. |
|
||
首页