MOOC学习场景下融合视频语义的多模态情绪识别
Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios
AI总结:
针对MOOC场景下现有情绪识别研究忽略教学视频语义影响的问题,提出融合视频语义、眼动与PPG信号的多模态情绪识别方法,显著提升识别性能,为相关研究提供新视角与参考。
摘要:
在大规模开放在线课程(MOOC)学习场景中,教学视频的语义信息对学习者的情绪状态具有关键影响。学习者主要通过观看教学视频获取知识,视频中的语义信息直接影响学习者的情绪状态。然而,鲜有研究关注教学视频的语义信息对学习者情绪状态的潜在影响。为深入探究视频语义信息对学习者情绪的影响,本文创新性地提出一种融合视频语义信息与生理信号的多模态情绪识别方法。我们通过预训练大语言模型(LLM)生成视频描述,以获取教学视频的高层语义信息。采用交叉注意力机制进行模态交互,将语义信息与眼动信号及光电容积脉搏波(PPG)信号融合,得到包含三种模态关键信息的特征。通过情绪分类器实现对学习者情绪状态的准确识别。实验结果表明,本文方法在情绪识别性能上得到显著提升,为MOOC学习场景下的情绪识别研究提供了新视角与高效方法。本文提出的方法不仅有助于更深入地理解教学视频对学习者情绪状态的影响,也为未来MOOC学习场景下的情绪识别研究提供了有益参考。
英文摘要:
In the Massive Open Online Courses (MOOC) learning scenario, the semantic information of instructional videos has a crucial impact on learners' emotional state. Learners mainly acquire knowledge by watching instructional videos, and the semantic information in the videos directly affects learners' emotional states. However, few studies have paid attention to the potential influence of the semantic information of instructional videos on learners' emotional states. To deeply explore the impact of video semantic information on learners' emotions, this paper innovatively proposes a multimodal emotion recognition method by fusing video semantic information and physiological signals. We generate video descriptions through a pre-trained large language model (LLM) to obtain high-level semantic information about instructional videos. Using the cross-attention mechanism for modal interaction, the semantic information is fused with the eye movement and PhotoPlethysmoGraphy (PPG) signals to obtain the features containing the critical information of the three modes. The accurate recognition of learners' emotional states is realized through the emotion classifier. The experimental results show that our method has significantly improved emotion recognition performance, providing a new perspective and efficient method for emotion recognition research in MOOC learning scenarios. The method proposed in this paper not only contributes to a deeper understanding of the impact of instructional videos on learners' emotional states but also provides a beneficial reference for future research on emotion recognition in MOOC learning scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。