数学教育中的自动化反馈：开放式回答的大语言模型比较分析
Automated Feedback in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses
AI总结：
本研究比较了Llama、SBERT-Canberra和GPT4三种大语言模型在数学教育中对开放式问题提供自动化评分与反馈的效果，发现微调模型与零样本方法各有优势，为个性化学习系统的发展提供了参考。
摘要：
反馈在提升学习成果方面的有效性在教育数据挖掘（EDM）中已有充分记录。以往的多项研究探索了提升反馈有效性的方法。大语言模型（LLMs）的最新发展扩展了其在增强自动化反馈系统方面的效用。本研究旨在探索LLMs在促进数学教育自动化反馈方面的潜力。我们通过比较3个不同的模型：Llama、SBERT-Canberra和GPT4模型，考察了LLMs在评估学生回答方面的有效性。该评估要求模型针对学生对开放式数学问题的回答，提供定量评分和定性反馈。我们采用了专为数学优化的Llama版本Mistral，并利用中学生数学问题的学生回答和教师手写反馈数据集，对该模型进行微调以评估学生回答。SBERT模型的训练也采用了类似方法，而GPT4模型则使用了零样本学习方法。我们利用2名教师的评判来评估模型在评分准确性和反馈质量方面的表现。教师们使用共享的评分标准来评估生成反馈的准确性和相关性。我们对模型性能进行了定量和定性分析。通过提供这些方法的详细比较，本研究旨在推进自动化反馈系统的持续发展，并为利用生成式LLMs创造更个性化学习体验概述潜在的未来方向。
英文摘要：
The effectiveness of feedback in enhancing learning outcomes is well documented within Educational Data Mining (EDM). Various prior research has explored methodologies to enhance the effectiveness of feedback. Recent developments in Large Language Models (LLMs) have extended their utility in enhancing automated feedback systems. This study aims to explore the potential of LLMs in facilitating automated feedback in math education. We examine the effectiveness of LLMs in evaluating student responses by comparing 3 different models: Llama, SBERT-Canberra, and GPT4 model. The evaluation requires the model to provide both a quantitative score and qualitative feedback on the student's responses to open-ended math problems. We employ Mistral, a version of Llama catered to math, and fine-tune this model for evaluating student responses by leveraging a dataset of student responses and teacher-written feedback for middle-school math problems. A similar approach was taken for training the SBERT model as well, while the GPT4 model used a zero-shot learning approach. We evaluate the model's performance in scoring accuracy and the quality of feedback by utilizing judgments from 2 teachers. The teachers utilized a shared rubric in assessing the accuracy and relevance of the generated feedback. We conduct both quantitative and qualitative analyses of the model performance. By offering a detailed comparison of these methods, this study aims to further the ongoing development of automated feedback systems and outlines potential future directions for leveraging generative LLMs to create more personalized learning experiences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
