arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节（Veterans Day），12 日无更新
arXiv 2409.18596cs.AIcs.CLcs.LG
收藏

ASAG2024：一个用于简答题评分的联合基准

ASAG2024: A Combined Benchmark for Short Answer Grading

Zurich University of Applied Sciences（苏黎世应用科技大学）

Gérôme Meyer, Philip Breuer, Jonathan Fürst

备注：
Accepted at SIGCSE-Virtual 2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出联合基准ASAG2024，整合七个简答题评分数据集以评估自动评分系统泛化能力，实验表明基于LLM的方法虽创新高但仍不及人类水平，为人机协同评分研究指明方向。

摘要：

开放式问题比封闭式问题能测试更透彻的理解能力，通常是更受青睐的评估方法。然而，开放式问题评分繁琐且易受个人偏见影响。因此，人们一直致力于通过自动化加快评分过程。简答题评分（SAG）系统旨在自动为学生答案打分。尽管SAG方法和能力有所增长，但目前仍缺乏跨不同学科、评分量表和分布的综合简答题评分基准。因此，难以评估当前自动评分方法的泛化能力。在这项初步工作中，我们引入了联合ASAG2024基准，以促进自动评分系统的比较。该基准将七个常用的简答题评分数据集整合为统一的结构和评分量表。针对该基准，我们评估了一系列近期的SAG方法，结果表明，尽管基于LLM的方法达到了新的高分，但仍远未达到人类水平。这为未来关于人机协同SAG系统的研究开辟了途径。

英文摘要：

Open-ended questions test a more thorough understanding than closed-ended questions and are often a preferred assessment method. However, open-ended questions are tedious to grade and subject to personal bias. Therefore, there have been efforts to speed up the grading process through automation. Short Answer Grading (SAG) systems aim to automatically score students' answers. Despite growth in SAG methods and capabilities, there exists no comprehensive short-answer grading benchmark across different subjects, grading scales, and distributions. Thus, it is hard to assess the capabilities of current automated grading methods in terms of their generalizability. In this preliminary work, we introduce the combined ASAG2024 benchmark to facilitate the comparison of automated grading systems. Combining seven commonly used short-answer grading datasets in a common structure and grading scale. For our benchmark, we evaluate a set of recent SAG methods, revealing that while LLM-based approaches reach new high scores, they still are far from reaching human performance. This opens up avenues for future research on human-machine SAG systems.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑