利用大语言模型测量教学
Measuring Teaching with LLMs
AI总结：
本文提出基于句级嵌入的定制大语言模型测量教学质量，通过数据高效训练和五种句嵌入的系统评估，实现超人类评分相关性，并揭示模型更依赖课堂层面特征而非孤立语句，为AI驱动的可扩展教学反馈提供了新方法。
摘要：
客观且可扩展的教学质量测量是教育领域长期存在的挑战。尽管大语言模型（LLMs）具有潜力，但通用模型在可靠地应用复杂、真实的课堂观察工具方面一直存在困难。本文使用基于句级嵌入构建的定制大语言模型，这种架构比传统的子词分词更适合课堂转录文本的长篇、解释性特征。我们在一种旨在防止过拟合的数据高效训练机制下，系统评估了五种不同的句嵌入。结果表明，这些专用模型能够达到人类水平甚至超人类的表现，与专家人工评分的相关性超过0.65，并超过了平均人与人之间的评分者相关性。此外，通过对标注上下文窗口的分析，我们发现更先进的模型——即那些与人类判断更一致的模型——将更大比例的分数变化归因于课堂层面的特征，而非孤立的语句，这对单轮标注范式的充分性提出了挑战。最后，为了评估外部效度，我们发现聚合的模型分数与教师增值测量指标一致，表明它们捕捉到了与学生学业学习相关的特征。然而，这一趋势在单个条目层面并不成立，说明虽然模型学到了有用的信号，但尚未实现完全泛化。本研究为人工智能驱动的教学测量建立了一种可行且强大的新方法，为教育者发展提供可扩展、可靠且有效的反馈开辟了道路。
英文摘要：
Objective and scalable measurement of teaching quality is a persistent challenge in education. While Large Language Models (LLMs) offer potential, general-purpose models have struggled to reliably apply complex, authentic classroom observation instruments. This paper uses custom LLMs built on sentence-level embeddings, an architecture better suited for the long-form, interpretive nature of classroom transcripts than conventional subword tokenization. We systematically evaluate five different sentence embeddings under a data-efficient training regime designed to prevent overfitting. Our results demonstrate that these specialized models can achieve human-level and even super-human performance with expert human ratings above 0.65 and surpassing the average human-human rater correlation. Further, through analysis of annotation context windows, we find that more advanced models-those better aligned with human judgments-attribute a larger share of score variation to lesson-level features rather than isolated utterances, challenging the sufficiency of single-turn annotation paradigms. Finally, to assess external validity, we find that aggregate model scores align with teacher value-added measures, indicating they are capturing features relevant to student learning. However, this trend does not hold at the individual item level, suggesting that while the models learn useful signals, they have not yet achieved full generalization. This work establishes a viable and powerful new methodology for AI-driven instructional measurement, offering a path toward providing scalable, reliable, and valid feedback for educator development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
