arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.15201cs.HC
收藏

利用语言、韵律和面部线索自动检测协作学习中的困惑与冲突

Automatically Detecting Confusion and Conflict During Collaborative Learning Using Linguistic, Prosodic, and Facial Cues

Yingbo Ma, Yukyeong Song, Mehmet Celepkolu, Kristy Elizabeth Boyer, Eric Wiebe, Collin F. Lynch, Maya Israel

AI总结:

提出融合语言、韵律与面部线索的多模态深度学习框架,基于38名小学生协作编程数据检测困惑与冲突,发现语义、音高和面部表情组合效果最佳。

摘要:

在协作学习过程中,困惑和冲突会自然出现。然而,持续的困惑或冲突有可能引发挫败感,并显著阻碍学习者的表现。对困惑和冲突进行早期自动检测,将使我们能够支持早期干预,从而改善学生的协作学习体验和成果。尽管已有大量研究对独自学习中的困惑进行建模,但协作学习领域仍需进一步研究。本文提出了一个多模态机器学习框架,用于自动检测协作学习中的困惑和冲突。我们使用了来自38名小学学习者的数据,这些学习者在教室中协作完成了一系列编程任务。我们利用从学习者协作对话中自动提取的特征训练深度多模态学习模型,以检测困惑和冲突,这些特征包括:(1)语言派生特征,包括TF-IDF、词汇语义和情感;(2)音频派生特征,包括声学-韵律特征;(3)视频派生特征,包括视线、头部姿态和面部表情。结果表明,结合语义、音高和面部表情的多模态模型检测困惑和冲突的准确率最高,优于所有单模态模型。我们还发现,韵律线索对冲突更具预测性,而面部线索对困惑更具预测性。本研究有助于协作学习过程的自动化建模,以及开发实时自适应支持,以增强学习者在课堂环境中的协作学习体验。

英文摘要:

During collaborative learning, confusion and conflict emerge naturally. However, persistent confusion or conflict have the potential to generate frustration and significantly impede learners' performance. Early automatic detection of confusion and conflict would allow us to support early interventions which can in turn improve students' experience with and outcomes from collaborative learning. Despite the extensive studies modeling confusion during solo learning, there is a need for further work in collaborative learning. This paper presents a multimodal machine-learning framework that automatically detects confusion and conflict during collaborative learning. We used data from 38 elementary school learners who collaborated on a series of programming tasks in classrooms. We trained deep multimodal learning models to detect confusion and conflict using features that were automatically extracted from learners' collaborative dialogues, including (1) language-derived features including TF-IDF, lexical semantics, and sentiment, (2) audio-derived features including acoustic-prosodic features, and (3) video-derived features including eye gaze, head pose, and facial expressions. Our results show that multimodal models that combine semantics, pitch, and facial expressions detected confusion and conflict with the highest accuracy, outperforming all unimodal models. We also found that prosodic cues are more predictive of conflict, and facial cues are more predictive of confusion. This study contributes to the automated modeling of collaborative learning processes and the development of real-time adaptive support to enhance learners' collaborative learning experience in classroom contexts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑