使用大型语言模型为学生解题过程解释分配部分学分：达到人类水平准确率的评分、评分置信度指数及个性化学生反馈
Using Large Language Models to Assign Partial Credit to Students' Explanations of Problem-Solving Process: Grade at Human Level Accuracy with Grading Confidence Index and Personalized Student-facing Feedback
AI总结：
本研究验证了用GPT-4o对物理题解题过程解释进行部分学分评分的可行性，其评分准确率达人类水平，还提出评分置信度指数优化评分监督，并能生成个性化学生反馈，降低评分成本。
摘要：
本研究探讨了使用大型语言模型（尤其是GPT-4o）对大量学生针对入门级物理题的书面解题过程解释进行部分学分评分的可行性和潜在优势。学生在课堂考试中被要求写下解决1道概念题和2道计算题时的推理过程文字解释，这些解释随后根据包含3个条目的评分标准进行评分，每个条目按二元（1或0）评分。研究首先证明，在不提供示例和参考答案的情况下，使用GPT-4o进行机器评分能与人工评分可靠一致，一致率达70%-80%，等于或高于两名人工评分者之间的一致率。实现该准确率的两个关键方法为：1. 为每个评分条目添加针对初始机器评分错误的解释性语言；2. 运行5次评分过程并取最频繁的结果。接着，研究表明通过香农熵衡量的5次机器评分结果的变异性可作为评分置信度指数，使教师仅需审查10%-15%的答卷就能识别约40%的潜在错误评分。最后，研究显示使用GPT-4o生成部分学分评分结果的清晰解释十分简便，这些解释可作为学生反馈，帮助学生理解成绩并在必要时提出异议。两名经验丰富的教师对生成的几乎所有反馈信息的5分制评分均为3分及以上。整个评分和反馈生成过程每100份学生答卷成本约5美元，表明结合机器评分与人工输入和监督的自动化方法对劳动密集型评分过程具有巨大应用前景。
英文摘要：
This study examines the feasibility and potential advantages of using large language models, in particular GPT-4o, to perform partial credit grading of large numbers of student written responses to introductory level physics problems. Students were instructed to write down verbal explanations of their reasoning process when solving one conceptual and two numerical calculation problems on in class exams. The explanations were then graded according to a 3-item rubric with each item grades as binary (1 or 0). We first demonstrate that machine grading using GPT-4o with no examples nor reference answer can reliably agree with human graders on 70%-80% of all cases, which is equal to or higher than the level at which two human graders agree with each other. Two methods are essential for achieving this level of accuracy: 1. Adding explanation language to each rubric item that targets the errors of initial machine grading. 2. Running the grading process 5 times and taking the most frequent outcome. Next, we show that the variation in outcomes across 5 machine grading attempts as measured by the Shannon Entropy can serve as a grading confidence index, allowing a human instructor to identify ~40% of all potentially incorrect gradings by reviewing just 10 - 15% of all responses. Finally, we show that it is straightforward to use GPT-4o to write clear explanations of the partial credit grading outcomes. Those explanations can be used as feedback for students, which will allow students to understand their grades and raise different opinions when necessary. Almost all feedback messages generated were rated 3 or above on a 5-point scale by two experienced instructors. The entire grading and feedback generating process cost roughly $5 per 100 student answers, which shows immense promise for automating labor-intensive grading process by a combination of machine grading with human input and supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
