评估大语言模型在课堂对话分析中的表现
Evaluating Large Language Models in Analysing Classroom Dialogue
AI总结：
本研究针对传统课堂对话定性分析的高成本问题，用定制化GPT-4分析中学数语课堂对话，对比人工编码发现其大幅省时且编码一致性高，展现LLM在教学评估中的应用潜力。
摘要：
本研究探讨大语言模型（Large Language Models, LLMs），具体为GPT-4，在课堂对话分析中的应用，该任务是教学诊断与质量提升的关键研究方向。鉴于教育研究中传统定性方法兼具知识密集型与劳动密集型的特点，本研究探究LLM简化并优化分析流程的潜力。研究采用来自一所中学的数据集，涵盖数学与语文课堂的对话内容。这些对话先由教育专家进行人工编码，随后使用定制化GPT-4模型开展分析。本研究重点对比人工标注结果与GPT-4的输出，评估其在教育对话分析中的效能，具体评估指标包括人类编码者与GPT-4之间的时间效率、编码者间一致性与编码者间信度。结果显示，GPT-4可大幅节省时间，且模型与人类编码者的编码结果高度一致，仅在特定编码上存在部分差异。这些发现凸显了LLM在教学评估与教学辅助方面的巨大潜力。
英文摘要：
This study explores the application of Large Language Models (LLMs), specifically GPT-4, in the analysis of classroom dialogue, a crucial research task for both teaching diagnosis and quality improvement. Recognizing the knowledge-intensive and labor-intensive nature of traditional qualitative methods in educational research, this study investigates the potential of LLM to streamline and enhance the analysis process. The study involves datasets from a middle school, encompassing classroom dialogues across mathematics and Chinese classes. These dialogues were manually coded by educational experts and then analyzed using a customised GPT-4 model. This study focuses on comparing manual annotations with the outputs of GPT-4 to evaluate its efficacy in analyzing educational dialogues. Time efficiency, inter-coder agreement, and inter-coder reliability between human coders and GPT-4 are evaluated. Results indicate substantial time savings with GPT-4, and a high degree of consistency in coding between the model and human coders, with some discrepancies in specific codes. These findings highlight the strong potential of LLM in teaching evaluation and facilitation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
