利用大语言模型与词袋模型自动评估课堂教学支持：将全局预测与具体反馈关联起来
Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback
AI总结：
本研究提出结合LLMs与BoW的机器学习架构，自动估算CLASS教学支持得分，准确率接近人类评分者信度，还可通过话语级可视化提供可解释的教学反馈。
摘要：
为向教师提供更具体、更频繁且可落地的教学反馈，本研究探索如何利用大语言模型（LLMs）估算“课堂评估评分系统（CLassroom Assessment Scoring System，CLASS）”的“教学支持”领域得分，CLASS是一种被广泛使用的课堂观察协议。我们设计了一套机器学习架构，可采用Meta的Llama2的零样本提示、和/或经典的词袋（Bag of Words，BoW）模型，对教师语音的单条话语（由OpenAI的Whisper自动转录）进行分类，判断其是否包含教学支持行为。随后，将这些话语层面的判断结果在15分钟的观察时段内聚合，以估算全局CLASS得分。在两个标注了CLASS编码的幼儿班与学前班课堂数据集上的实验表明：（1）采用所提方法自动估算CLASS教学支持得分的准确率（皮尔逊相关系数R最高达0.48）接近人类评分者间信度（最高R=0.55）；（2）在该任务上，LLMs的准确率通常略高于BoW，但性能最优的模型往往结合了从LLM和BoW中提取的特征；（3）在单条话语分类任务上，自动化方法与人类水平的判断相比仍有提升空间。最后，（4）我们展示了如何在话语层面可视化模型输出，为教师提供可解释的反馈，说明哪些话语与特定CLASS维度的正相关或负相关性最强。
英文摘要：
With the aim to provide teachers with more specific, frequent, and actionable feedback about their teaching, we explore how Large Language Models (LLMs) can be used to estimate ``Instructional Support'' domain scores of the CLassroom Assessment Scoring System (CLASS), a widely used observation protocol. We design a machine learning architecture that uses either zero-shot prompting of Meta's Llama2, and/or a classic Bag of Words (BoW) model, to classify individual utterances of teachers' speech (transcribed automatically using OpenAI's Whisper) for the presence of Instructional Support. Then, these utterance-level judgments are aggregated over a 15-min observation session to estimate a global CLASS score. Experiments on two CLASS-coded datasets of toddler and pre-kindergarten classrooms indicate that (1) automatic CLASS Instructional Support estimation accuracy using the proposed method (Pearson $R$ up to $0.48$) approaches human inter-rater reliability (up to $R=0.55$); (2) LLMs generally yield slightly greater accuracy than BoW for this task, though the best models often combined features extracted from both LLM and BoW; and (3) for classifying individual utterances, there is still room for improvement of automated methods compared to human-level judgments. Finally, (4) we illustrate how the model's outputs can be visualized at the utterance level to provide teachers with explainable feedback on which utterances were most positively or negatively correlated with specific CLASS dimensions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
