arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.02450cs.LGcs.AIcs.CL
收藏

我们是否需要前沿模型来验证数学证明?

Do We Need Frontier Models to Verify Mathematical Proofs?

University of Pennsylvania(宾夕法尼亚大学)

Aaditya Naik, Guruprerana Shabadi, Rajeev Alur, Mayur Naik

AI总结:

研究评估了开源和前沿LLM在验证数学证明中的表现,发现较小模型在准确性上接近前沿模型,但一致性较差,通过定制提示词提升性能。

摘要:

训练、后训练和推理时的方法进步使前沿推理模型在数学竞赛中获奖并解决难题。验证这些模型响应需要检查自然语言证明中的错误。LLM裁判被越来越多用于评估证明。尽管验证被认为比生成更容易,但可靠验证实际需要哪些模型能力?我们系统评估了四个开源和两个前沿LLM,在人类评分的自然语言证明数据集上。我们考虑两个关键指标:验证准确性与自一致性(相同证明重复判断的一致性率)。观察到较小的开源模型在准确性上仅比前沿模型低约10%,但在一致性上低约25%。此外,验证准确性对提示选择敏感。我们发现较小模型确实具备验证前沿模型水平证明的能力,但难以通过通用提示可靠激发这些能力。通过LLM引导的提示搜索,我们合成了一组专用提示,克服较小模型的特定失败模式,提升性能达9.1%(准确性)和15.9%(自一致性)。这些提升在所有模型和数据集上实现,使如Qwen3.5-35B等模型能与Gemini 3.1 Pro等前沿模型在证明验证上媲美。

英文摘要:

Advances in training, post-training, and inference-time methods have enabled frontier reasoning models to win gold medals in math competitions and settle challenging open problems. Gaining trust in the responses of these models requires that natural language proofs be checked for errors. LLM judges are increasingly being adopted to meet the growing demand for evaluating such proofs. While verification is considered easier than generation, what model capability does reliable verification actually require? We systematically evaluate four open-source and two frontier LLMs on datasets of human-graded natural language proofs of competition-level problems. We consider two key metrics: verifier accuracy and self-consistency (the rate of agreement across repeated judgments on the same proof). We observe that smaller open-source models are only up to ~10% behind frontier models in accuracy but they are up to ~25% more inconsistent. Furthermore, we see that verifier accuracy is sensitive to prompt choice across all models. We then demonstrate that the smaller models, in fact, do possess the mathematical capabilities to verify proofs at the level of frontier models, but they struggle to reliably elicit these capabilities with general judging prompts. Through an LLM-guided prompt search, we synthesize an ensemble of specialized prompts that overcome the specific failure modes of smaller models, boosting their performance by up to 9.1% in accuracy and 15.9% in self-consistency. These gains are realized across models and datasets, allowing models like Qwen3.5-35B to perform on par with frontier models such as Gemini 3.1 Pro for proof verification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑