评估前沿大语言模型在博士级数学推理中的表现：一个关于随机算法理论教材的基准测试
Evaluating Frontier LLMs on PhD-Level Mathematical Reasoning: A Benchmark on a Textbook in Theoretical Computer Science about Randomized Algorithms
AI总结：
本文评估了前沿大语言模型在博士级数学推理中的表现，通过随机算法教材的基准测试，发现顶级模型在准确性上表现优异，但可靠性存在显著差异。
摘要：
大型语言模型（LLMs）的快速发展已推动了自动化数学推理和科学发现的重大突破。Georgiev、Gómez-Serrano、Tao和Wagner [GGSTW+25] 证明人工智能系统能够探索新构造并改进现有界限，展示了LLMs在加速数学发现方面的增长潜力。同样，Bubeck等人 [BCE+25] 表明GPT-5可以有意义地贡献于科学流程，从提出假设到生成证明和分析。尽管有这些进展，对这些模型在经典、研究生级数学理论上的严格评估仍然是必要的，以了解其基础推理能力。在本文中，我们提出了四个前沿模型：GPT-5-Thinking、Gemini-3-Pro、Claude-Sonnet-4.5-Thinking和Grok-4，针对Motwani和Raghavan [MR95] 的经典课程《随机算法》的全面基准测试。我们让每个模型生成一系列引理和练习的正式LaTeX证明。我们发现，尽管顶级模型（Gemini和Claude）在准确性方面达到高比率（约66%），显示出对概率方法和形式逻辑的牢固掌握，其他模型在一致性方面显著滞后（约40%）。我们提供了生成证明的定性分析，突显了简洁性、幻觉率和逻辑结构之间的差异。我们的结果表明，尽管前沿模型已达到适合研究生级教学辅助和形式化的熟练度阈值，但它们在严格数学推导中的可靠性仍存在显著差异。代码和完整的LLM生成响应已开源并公开可用，网址为https://github.com/magiclinux/math_benchmark_probability。
英文摘要：
The rapid advancement of large language models (LLMs) has led to significant breakthroughs in automated mathematical reasoning and scientific discovery. Georgiev, G${ó}$mez-Serrano, Tao, and Wagner [GGSTW+25] demonstrate that AI systems can explore new constructions and improve existing bounds, illustrating the growing potential of LLMs to accelerate mathematical discovery. Similarly, Bubeck et al. [BCE+25] show that GPT-5 can meaningfully contribute to scientific workflows, from proposing hypotheses to generating proofs and analyses. Despite these advances, a rigorous evaluation of these models on canonical, graduate-level mathematical theory remains necessary to understand their baseline reasoning capabilities. In this paper, we present a comprehensive benchmark of four frontier models: GPT-5-Thinking, Gemini-3-Pro, Claude-Sonnet-4.5-Thinking, and Grok-4 against the classic curriculum of Randomized Algorithms by Motwani and Raghavan [MR95]. We tasked each model with generating formal LaTeX proofs for a series of lemmas and exercises spanning the textbook. We find that while the top-tier models (Gemini, and Claude) achieve a high accuracy rate (approx. 66%), demonstrating a robust grasp of probabilistic method and formal logic, other models lag significantly in consistency (approx. 40%). We provide a qualitative analysis of the generated proofs, highlighting differences in conciseness, hallucination rates, and logical structure. Our results suggest that while frontier models have reached a threshold of proficiency suitable for graduate-level pedagogical assistance and formalization, significant variance exists in their reliability for rigorous mathematical derivation. The code and the full set of LLM-generated responses are open-sourced and publicly available at https://github.com/magiclinux/math_benchmark_probability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
