RefGrader：使用智能体工作流自动评分数学竞赛证明
RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows
AI总结：
针对LLM在数学竞赛证明评分中部分分数分配校准不足的问题，提出基于智能体工作流的RefGrader方法，通过提取分析参考解并自动生成评分标准，实现更公平、与人类评分更一致的自动评分。
摘要：
最先进（SOTA）的大语言模型（LLM）已从在基于证明的奥林匹克竞赛问题上挣扎，发展到能解决IMO 2025的大多数问题，据报道，领先系统能处理6个问题中的5个。鉴于这一进展，我们评估这些模型在评分证明方面的能力：检测错误、判断其严重性，并在二元正确性之外分配公平的分数。我们使用一个由90个Gemini 2.5 Pro生成的解决方案组成的语料库来研究证明分析能力，我们按1-4分制进行评分并附带详细的错误注释，同时使用MathArena上针对IMO/USAMO 2025的解决方案集（按0-7分制评分）。我们的分析表明，模型能可靠地标记不正确（包括细微不正确）的解决方案，但在部分分数的分配上表现出校准差距。为解决这一问题，我们引入了智能体工作流，该工作流提取并分析参考解决方案，并自动推导出针对具体问题的评分标准，用于多步骤评分过程。我们实例化并比较了评分工作流的不同设计选择，并评估了它们的权衡。在我们的注释语料库和MathArena上，我们提出的工作流与人类评分的一致性更高，并且在各项指标上对部分分数的处理更加一致。我们发布所有代码、数据和提示/日志，以促进未来的研究。
英文摘要：
State-of-the-art (SOTA) LLMs have progressed from struggling on proof-based Olympiad problems to solving most of the IMO 2025 problems, with leading systems reportedly handling 5 of 6 problems. Given this progress, we assess how well these models can grade proofs: detecting errors, judging their severity, and assigning fair scores beyond binary correctness. We study proof-analysis capabilities using a corpus of 90 Gemini 2.5 Pro-generated solutions that we grade on a 1-4 scale with detailed error annotations, and on MathArena solution sets for IMO/USAMO 2025 scored on a 0-7 scale. Our analysis shows that models can reliably flag incorrect (including subtly incorrect) solutions but exhibit calibration gaps in how partial credit is assigned. To address this, we introduce agentic workflows that extract and analyze reference solutions and automatically derive problem-specific rubrics for a multi-step grading process. We instantiate and compare different design choices for the grading workflows, and evaluate their trade-offs. Across our annotated corpus and MathArena, our proposed workflows achieve higher agreement with human grades and more consistent handling of partial credit across metrics. We release all code, data, and prompts/logs to facilitate future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
