Advances in training, post-training, and inference-time methods have enabled frontier reasoning models to win gold medals in math competitions and settle challenging open problems. Gaining trust in the responses of these models requires that natural language proofs be checked for errors. LLM judges are increasingly being adopted to meet the growing demand for evaluating such proofs. While verification is considered easier than generation, what model capability does reliable verification actually require? We systematically evaluate four open-source and two frontier LLMs on datasets of human-graded natural language proofs of competition-level problems. We consider two key metrics: verifier accuracy and self-consistency (the rate of agreement across repeated judgments on the same proof). We observe that smaller open-source models are only up to ~10% behind frontier models in accuracy but they are up to ~25% more inconsistent. Furthermore, we see that verifier accuracy is sensitive to prompt choice across all models. We then demonstrate that the smaller models, in fact, do possess the mathematical capabilities to verify proofs at the level of frontier models, but they struggle to reliably elicit these capabilities with general judging prompts. Through an LLM-guided prompt search, we synthesize an ensemble of specialized prompts that overcome the specific failure modes of smaller models, boosting their performance by up to 9.1% in accuracy and 15.9% in self-consistency. These gains are realized across models and datasets, allowing models like Qwen3.5-35B to perform on par with frontier models such as Gemini 3.1 Pro for proof verification.
Evaluating Frontier LLMs on PhD-Level Mathematical Reasoning: A Benchmark on a Textbook in Theoretical Computer Science about Randomized Algorithms
Wyoming Seminary(怀俄明神学院);San Jose State University(圣何塞州立大学);Guilin University of Electronic Technology(桂林电子科技大学);University of California, Berkeley(加州大学伯克利分校);Northeastern University(东北大学);Guangdong University of Technology(广东工业大学)
FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levels
Westlake Institute for Advanced Study, Westlake University(西拉丘学院先进研究院,西拉丘大学);Peking University(北京大学);New Cornerstone Science Laboratory, School of Mathematical Sciences, Peking University(新基石科学实验室,北京大学数学科学学院);Center for Machine Learning Research, Peking University(机器学习研究中心,北京大学);Center for Intelligent Computing, Great Bay Institute for Advanced Study, Great Bay University(智能计算中心,大湾先进研究院,大湾大学)
An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Department of Computer Science University of Illinois Urbana–Champaign(计算机科学系伊利诺伊大学厄巴纳-香槟分校);Department of Computer Science Stanford University(计算机科学系斯坦福大学)
RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning
Artificial Intelligence Thrust, Information Hub, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China(人工智能 thrust,信息中心,香港科学与技术大学(广州),广州,中国)
Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning
The Wharton School, University of Pennsylvania(宾夕法尼亚大学沃顿商学院);Carnegie Mellon University(卡内基梅隆大学);Harvard Business School, Harvard University(哈佛大学哈佛商学院)
A Survey on Large Language Models for Mathematical Reasoning
National Key Laboratory for Novel Software Technology(国家新型软件技术重点实验室);School of Artificial Intelligence(人工智能学院);Nanyang Technological University(南洋理工大学);Skywork AI Singapore(Skywork AI新加坡);The Chinese University of Hong Kong Shenzhen(香港中文大学(深圳))
Putnam-like dataset summary: LLMs as mathematical competition contestants
Faculty of Mathematics, Informatics;Mechanics, University of Warsaw, Banacha 2, 02-097 Warsaw, Poland;Faculty of Mathematics;Computer Science, Nicolaus Copernicus University in Toru\'n, Chopina 12/18, 87-100 Toru\'n, Poland
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Xi’an Jiaotong-Liverpool University(西交利物浦大学);University of Liverpool(利物浦大学);University of Macau(澳门大学);HKUST(香港科技大学);Microsoft Research Asia(微软亚洲研究院);Duke Kunshan University(昆山杜克大学)
CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions
University of Massachusetts Lowell(马萨诸塞大学洛文分校);San Jose State University(圣何塞州立大学);Massachusetts Institute of Technology(麻省理工学院);Dartmouth College(达特茅斯学院);Amazon AGI(亚马逊人工智能研究院)
AIProver: Agentic Auto-Formalization of Mathematical Research via Certificate-Driven Evolving Harness
Georgia Institute of Technology(佐治亚理工学院);University of Pennsylvania(宾夕法尼亚大学);Foothill College(山麓学院);Rutgers University(罗格斯大学);Simon Fraser University(西蒙弗雷泽大学);University of Texas at Austin(德克萨斯大学奥斯汀分校)
Learn and Verify: A Framework for Rigorous Verification of Physics-Informed Neural Networks
Global Center for Science and Engineering(全球科学与工程中心);Waseda University(早稻田大学);Faculty of Engineering(工学院);Tokyo University of Agriculture and Technology(东京农业大学)
Adaptive Generate-Rank-Verify: Inference-Time Search with Costly Verification
University of Southern California(南加州大学);Northwestern University(西北大学);University of Chicago(芝加哥大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);Simons Institute for the Theory of Computing(Simons计算理论研究所);Data Science Institute at the University of Chicago(芝加哥大学数据科学研究所)
Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities
Agency for Science, Technology and Research (A*STAR)(科技研究局);Indian Institute of Technology (IIT) Goa(印度理工学院 Goa分校);Nanyang Technological University(南洋理工大学)