arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.13337cs.CLcs.AI
收藏

语言模型是少样本评分器

Language Models are Few-Shot Graders

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Utah State University(犹他州立大学)

Chenyan Zhao, Mariana Silva, Seth Poulsen

AI总结:

本文提出基于 LLM 的自动简答题评分流程,比较 GPT-4、GPT-4o 与 o1-preview,并验证 RAG 选择示例和评分量规可提升评分准确性。

摘要:

为学生作业提供评价是有效学习的关键环节,而将这一过程自动化可以显著减轻人工评分者的工作量。得益于大型语言模型(LLM)的进步,自动简答题评分(ASAG)系统为评估开放式学生回答并提供即时反馈提供了一种有前景的方案。本文提出了一种利用最先进 LLM 的 ASAG 流程。我们新的基于 LLM 的 ASAG 流程在相同数据集上取得了优于现有定制模型的性能。我们还比较了三个 OpenAI 模型——GPT-4、GPT-4o 和 o1-preview——的评分表现。结果表明,GPT-4o 在准确性与成本效益之间取得了最佳平衡;相比之下,o1-preview 虽然准确性更高,但误差方差更大,因此不太适合课堂实际使用。我们研究了在提示中加入由教师评分过的示例所产生的影响,采用了无示例、随机选择以及基于检索增强生成(RAG)的选择策略。研究发现,提供已评分示例能够提高评分准确性,而且基于 RAG 的选择优于随机选择。此外,融入评分量规可以通过提供结构化的评价标准来提高准确性。

英文摘要:

Providing evaluations to student work is a critical component of effective student learning, and automating its process can significantly reduce the workload on human graders. Automatic Short Answer Grading (ASAG) systems, enabled by advancements in Large Language Models (LLMs), offer a promising solution for assessing and providing instant feedback for open-ended student responses. In this paper, we present an ASAG pipeline leveraging state-of-the-art LLMs. Our new LLM-based ASAG pipeline achieves better performances than existing custom-built models on the same datasets. We also compare the grading performance of three OpenAI models: GPT-4, GPT-4o, and o1-preview. Our results demonstrate that GPT-4o achieves the best balance between accuracy and cost-effectiveness. On the other hand, o1-preview, despite higher accuracy, exhibits a larger variance in error that makes it less practical for classroom use. We investigate the effects of incorporating instructor-graded examples into prompts using no examples, random selection, and Retrieval-Augmented Generation (RAG)-based selection strategies. Our findings indicate that providing graded examples enhances grading accuracy, with RAG-based selection outperforming random selection. Additionally, integrating grading rubrics improves accuracy by offering a structured standard for evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑