arXiv 2505.072472025-05-15cs.CLcs.AI

SAS-Bench：一个用于评估大语言模型短答案评分的细粒度基准

SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models

Peking University（北京大学）；Fuzhou University（福州市大学）；Hunan University（湖南大学）；Shanghai AI Laboratory（上海人工智能实验室）

Peichao Lai, Kexuan Zhang, Yi Lin, Linyihan Zhang, Feiyang Ye, Jinhao Yan, Yanwei Xu, Conghui He, Yilei Wang, Wentao Zhang, Bin Cui

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出面向大语言模型短答案评分的细粒度基准 SAS-Bench，并发布专家标注数据集，通过多模型实验揭示科学题评分挑战及少样本提示的增益。

摘要：

主观答案评分（Subjective Answer Grading, SAG）在教育、标准化考试和自动评估系统中发挥着关键作用，尤其适用于评估短答案评分（Short Answer Scoring, SAS）中的简短作答。然而，现有方法通常只能给出粗粒度分数，且缺乏详细推理。尽管大语言模型（LLM）已展现出作为零样本评估器的潜力，但它们仍容易受到偏见影响，与人类判断不一致，并且评分决策的透明度有限。为克服这些局限，我们提出 SAS-Bench，这是一个专门为基于 LLM 的 SAS 任务设计的基准。SAS-Bench 提供细粒度、分步骤的评分、专家标注的错误类别，以及源自真实学科考试的多种题型。该基准有助于详细评估模型的推理过程和可解释性。我们还发布了一个开源数据集，包含 1,030 个问题和 4,109 份学生作答，每份均由领域专家标注。此外，我们使用多种 LLM 进行了全面实验，识别出科学相关问题评分中的主要挑战，并强调少样本提示在提升评分准确性方面的有效性。本研究为开发更稳健、公平且具有教育意义的基于 LLM 的评估系统提供了宝贵洞见。

英文摘要：

Subjective Answer Grading (SAG) plays a crucial role in education, standardized testing, and automated assessment systems, particularly for evaluating short-form responses in Short Answer Scoring (SAS). However, existing approaches often produce coarse-grained scores and lack detailed reasoning. Although large language models (LLMs) have demonstrated potential as zero-shot evaluators, they remain susceptible to bias, inconsistencies with human judgment, and limited transparency in scoring decisions. To overcome these limitations, we introduce SAS-Bench, a benchmark specifically designed for LLM-based SAS tasks. SAS-Bench provides fine-grained, step-wise scoring, expert-annotated error categories, and a diverse range of question types derived from real-world subject-specific exams. This benchmark facilitates detailed evaluation of model reasoning processes and explainability. We also release an open-source dataset containing 1,030 questions and 4,109 student responses, each annotated by domain experts. Furthermore, we conduct comprehensive experiments with various LLMs, identifying major challenges in scoring science-related questions and highlighting the effectiveness of few-shot prompting in improving scoring accuracy. Our work offers valuable insights into the development of more robust, fair, and educationally meaningful LLM-based evaluation systems.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。