判断我们无法解决的内容：一种基于后果的方法用于无Oracle评估研究级数学
Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Math
AI总结：
本文提出基于后果的无Oracle评估方法，用于评估研究级数学问题的求解质量，通过测试候选方案在相关问题中的表现，有效提升评估准确性。
摘要：
最近在推理模型中的进展表明，生成合理尝试以解决研究级数学可能已近在咫尺，但验证仍是一个瓶颈，消耗稀缺的专家时间。我们假设一个有意义的解决方案应包含足够的方法级信息，当应用于相关问题的邻域时，应比错误的解决方案产生更好的下游性能。基于这一想法，我们提出了基于后果的效用，一种无Oracle的评估器，通过测试每个候选者作为上下文示例在解决可验证的相关问题中的价值来评分。我们的方法在一组原始的研究级数学问题上进行评估，每个问题配有一个专家撰写的解决方案和九个LLM生成的解决方案。值得注意的是，基于后果的效用在排名质量上始终优于奖励模型、生成奖励模型和LLM评估者。具体而言，对于GPT-OSS-120B，它将Acc@1从67.2提高到76.3，AUC从71.4提高到79.6，同样在GPT-OSS-20B上也产生了相似大的AUC提升（69.0到79.2）。此外，与LLM评估者相比，它还表现出更大的求解者-评估者差距，在底层求解器经常无法解决的实例上也保持了更强的正确-错误分离。
英文摘要：
Recent progress in reasoning models suggests that generating plausible attempts for research-level mathematics may be within reach, but verification remains a bottleneck, consuming scarce expert time. We hypothesize that a meaningful solution should contain enough method-level information that, when applied to a neighborhood of related questions, it should yield better downstream performance than incorrect solutions. Building on this idea, we propose \textbf{Consequence-Based Utility}, an oracle-free evaluator that scores each candidate by testing its value as an in-context exemplar in solving related yet verifiable questions. Our approach is evaluated on an original set of research-level math problems, each paired with one expert-written solution and nine LLM-generated solutions. Notably, Consequence-Based Utility consistently outperforms reward models, generative reward models, and LLM judges on ranking quality. Specifically, for GPT-OSS-120B, it improves Acc@1 from 67.2 to 76.3 and AUC from 71.4 to 79.6, with similarly large AUC gains on GPT-OSS-20B (69.0 to 79.2). Furthermore, compared to LLM-Judges, it also exhibits a larger solver-evaluator gap, maintaining a stronger correct-wrong separation even on instances where the underlying solver often fails to solve.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
