arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.01754cs.CLcs.AIcs.LG
收藏

LiveMathematicianBench: 一个用于数学家级推理的实时基准

LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches

Columbia University(哥伦比亚大学);Microsoft Research(微软研究院);University of Amsterdam(阿姆斯特丹大学)

Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, Nima Mesgarani

AI总结:

本文提出LiveMathematicianBench,一个基于近期arXiv论文的动态多选基准,用于评估大语言模型的数学推理能力,通过证明草图指导的干扰项生成机制,提升对真实理解的检测。

摘要:

数学推理是人类智能的标志,大语言模型(LLMs)能否有意义地进行数学推理仍然是人工智能和认知科学的核心问题。随着LLMs越来越多地融入科学工作流程,对其数学能力的严格评估成为实际需求。现有基准受限于合成环境和数据污染。我们提出了LiveMathematicianBench,一个动态多选基准,基于最近发表的arXiv论文,这些论文发布在模型训练截止日期之后。通过将评估基于新发表的定理,它提供了一个超越记忆模式的现实测试环境。该基准引入了十三类逻辑分类的定理类型(例如,蕴含、等价、存在、唯一性),使在推理形式上能够进行细粒度评估。它采用证明草图指导的干扰项生成管道,利用高层证明策略构建看似合理但无效的答案选项,反映误导的证明方向,增加对真实理解的敏感性。我们还引入了一种抗替换机制,以区分答案识别与实质推理。评估显示该基准远未饱和:Gemini-3.1-pro-preview最佳模型仅达到43.5%。在抗替换评估下,准确性急剧下降:GPT-5.4最高达到30.6%,而Gemini-3.1-pro-preview降至17.6%,低于20%的随机基线。双模式协议揭示证明草图访问带来了一致的准确性提升,表明模型可以利用高层证明策略进行推理。总体而言,LiveMathematicianBench为研究LLM在研究级数学推理中的可扩展、抗污染测试环境。

英文摘要:

Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a central question in artificial intelligence and cognitive science. As LLMs are increasingly integrated into scientific workflows, rigorous evaluation of their mathematical capabilities becomes a practical necessity. Existing benchmarks are limited by synthetic settings and data contamination. We present LiveMathematicianBench, a dynamic multiple-choice benchmark for research-level mathematical reasoning built from recent arXiv papers published after model training cutoffs. By grounding evaluation in newly published theorems, it provides a realistic testbed beyond memorized patterns. The benchmark introduces a thirteen-category logical taxonomy of theorem types (e.g., implication, equivalence, existence, uniqueness), enabling fine-grained evaluation across reasoning forms. It employs a proof-sketch-guided distractor pipeline that uses high-level proof strategies to construct plausible but invalid answer choices reflecting misleading proof directions, increasing sensitivity to genuine understanding over surface-level matching. We also introduce a substitution-resistant mechanism to distinguish answer recognition from substantive reasoning. Evaluation shows the benchmark is far from saturated: Gemini-3.1-pro-preview, the best model, achieves only 43.5%. Under substitution-resistant evaluation, accuracy drops sharply: GPT-5.4 scores highest at 30.6%, while Gemini-3.1-pro-preview falls to 17.6%, below the 20% random baseline. A dual-mode protocol reveals that proof-sketch access yields consistent accuracy gains, suggesting models can leverage high-level proof strategies for reasoning. Overall, LiveMathematicianBench offers a scalable, contamination-resistant testbed for studying research-level mathematical reasoning in LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑