arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.02165cs.AIcs.CL
收藏

一种基于大语言模型并具有人类水平评分标准优化能力的自动评分框架

A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimization

Michigan State University(密歇根州立大学)

Yucheng Chu, Hang Li, Kaiqi Yang, Harry Shomer, Hui Liu, Yasemin Copur-Gencturk, Jiliang Tang

AI总结:

本文提出多智能体自动简答题评分框架GradeOpt,利用大语言模型并引入反思者与优化者自动优化评分标准,在PCK和CK评分任务中显著提升评分准确性与人类评分一致性。

摘要:

开放式简答题(SAGs)在学习分析(LA)领域已被广泛认为是深入了解学习者作答情况的有力工具。然而,由于评分工作量大以及评估不一致等问题,开放式简答题在实践中往往面临挑战。随着自然语言处理(NLP)的最新进展,自动简答题评分(ASAG)为这些挑战提供了一种有前景的解决方案。尽管如此,当前的ASAG算法在泛化能力上往往受限,并且倾向于针对特定问题进行定制。在本文中,我们提出了一种统一的多智能体ASAG框架,名为GradeOpt,该框架利用大语言模型(LLMs)作为开放式简答题的评分者。更重要的是,GradeOpt在多智能体系统中引入了两个额外的基于LLM的智能体——反思者(reflector)和优化者(refiner)。这使得GradeOpt能够通过对其错误进行自我反思来自动优化原始评分标准。通过在具有挑战性的ASAG任务上进行实验,即对学科教学知识(PCK)和内容知识(CK)问题进行评分,GradeOpt在评分准确性和与人类评分者的行为一致性方面,相较于代表性基线方法均表现出优越的性能。最后,全面的消融研究证实了GradeOpt中各个设计组件的有效性。

英文摘要:

Open-ended short-answer questions (SAGs) have been widely recognized as a powerful tool for providing deeper insights into learners' responses in the context of learning analytics (LA). However, SAGs often present challenges in practice due to the high grading workload and concerns about inconsistent assessments. With recent advancements in natural language processing (NLP), automatic short-answer grading (ASAG) offers a promising solution to these challenges. Despite this, current ASAG algorithms are often limited in generalizability and tend to be tailored to specific questions. In this paper, we propose a unified multi-agent ASAG framework, GradeOpt, which leverages large language models (LLMs) as graders for SAGs. More importantly, GradeOpt incorporates two additional LLM-based agents - the reflector and the refiner - into the multi-agent system. This enables GradeOpt to automatically optimize the original grading guidelines by performing self-reflection on its errors. Through experiments on a challenging ASAG task, namely the grading of pedagogical content knowledge (PCK) and content knowledge (CK) questions, GradeOpt demonstrates superior performance in grading accuracy and behavior alignment with human graders compared to representative baselines. Finally, comprehensive ablation studies confirm the effectiveness of the individual components designed in GradeOpt.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑