QEDBENCH:量化大学水平数学证明自动评估中的对齐差距
QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
AI总结:
研究大语言模型自动评估的可靠性,引入QEDBench基准,通过对比特定课程评分标准与专家常识标准,测量与人类专家的对齐情况,揭示部分前沿评估器偏差及离散领域推理差距,还发布该基准用于评估改进AI裁判。
摘要:
随着大语言模型在基础基准测试中达到饱和,研究前沿已从生成转向自动评估的可靠性。我们证明,当应用于本科高年级到研究生早期水平的数学时,标准的“大语言模型作为评判”协议存在系统性对齐差距。为了量化这一点,我们引入了QEDBench,这是第一个大规模双评分对齐基准,通过将特定课程的评分标准与专家常识标准进行对比,系统地测量与人类专家在大学水平数学证明上的对齐情况。通过对1000多个小时的人工评估部署双评估矩阵(7名评判×5名求解器),我们发现某些前沿评估器,如Claude Opus 4.5、DeepSeek-V3、Qwen 2.5 Max和Llama 4 Maverick表现出显著的正偏差(平均分数膨胀分别高达+0.18、+0.20、+0.30、+0.36)。此外,我们发现离散领域存在关键的推理差距:虽然Gemini 3.0 Pro实现了领先性能(平均人工评估分数为0.91),但其他推理模型,如GPT-5 Pro和Claude Sonnet 4.5在离散领域的性能显著下降。具体而言,它们在离散数学中的平均人工评估分数降至0.72和0.63,在图论中降至0.74和0.50。除了这些研究结果,我们还发布了QEDBench作为评估和改进AI评判的公共基准。我们的基准在这个https URL上公开发布。
英文摘要:
As Large Language Models (LLMs) saturate elementary benchmarks, the research frontier has shifted from generation to the reliability of automated evaluation. We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics. To quantify this, we introduce QEDBench, the first large-scale dual-rubric alignment benchmark to systematically measure alignment with human experts on university-level math proofs by contrasting course-specific rubrics against expert common knowledge criteria. By deploying a dual-evaluation matrix (7 judges x 5 solvers) against 1,000+ hours of human evaluation, we reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias (up to +0.18, +0.20, +0.30, +0.36 mean score inflation, respectively). Furthermore, we uncover a critical reasoning gap in the discrete domain: while Gemini 3.0 Pro achieves state-of-the-art performance (0.91 average human evaluation score), other reasoning models like GPT-5 Pro and Claude Sonnet 4.5 see their performance significantly degrade in discrete domains. Specifically, their average human evaluation scores drop to 0.72 and 0.63 in Discrete Math, and to 0.74 and 0.50 in Graph Theory. In addition to these research results, we also release QEDBench as a public benchmark for evaluating and improving AI judges. Our benchmark is publicly published at https://github.com/qqliu/Yale-QEDBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。