近期的大语言模型（LLMs）已在长上下文场景中展现出多方面能力。尽管已有部分近期基准被开发用于评估LLMs的长上下文能力，但目前仍缺乏评估LLMs在长上下文下数学推理能力的基准，而这一能力对LLMs在真实场景中的应用至关重要。本文提出了MathHay，这是一款专为评估LLMs长上下文数学推理能力设计的自动化基准。与此前主要聚焦长文本内信息检索的Needle in a Haystack（大海捞针）等基准不同，MathHay要求模型同时具备信息检索能力与复杂数学推理能力。我们在MathHay上开展了大量实验，对8款性能顶尖的LLMs的长上下文数学推理能力进行评估。即便是表现最优的模型Gemini-1.5-Pro-002，在长上下文数学推理任务上仍存在困难，在128K tokens的设置下仅达到51.26%的准确率。这一结果凸显出模型在MathHay基准上仍有很大提升空间。
英文摘要：
Recent large language models (LLMs) have demonstrated versatile capabilities in long-context scenarios. Although some recent benchmarks have been developed to evaluate the long-context capabilities of LLMs, there is a lack of benchmarks evaluating the mathematical reasoning abilities of LLMs over long contexts, which is crucial for LLMs' application in real-world scenarios. In this paper, we introduce MathHay, an automated benchmark designed to assess the long-context mathematical reasoning capabilities of LLMs. Unlike previous benchmarks like Needle in a Haystack, which focus primarily on information retrieval within long texts, MathHay demands models with both information-seeking and complex mathematical reasoning abilities. We conduct extensive experiments on MathHay to assess the long-context mathematical reasoning abilities of eight top-performing LLMs. Even the best-performing model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights the significant room for improvement on the MathHay benchmark.
Toward Automated Robustness Evaluation of Mathematical Reasoning
Shanghai University of Finance and Economics（上海金融学院）；Ant Group（蚂蚁集团）；Southern University of Science and Technology（南方科技大学）；MoE Key Laboratory of Interdisciplinary Research of Computation and Economics（计算与经济学交叉学科研究教育部重点实验室）
FMC: Formalization of Natural Language Mathematical Competition Problems
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University, Beijing, China（多媒体信息处理国家重点实验室，计算机科学学院，PKU-Anker LLM实验室，北京大学，北京，中国）；Paul G. Allen School of Computer Science（保罗·G·阿伦计算机科学学校）；Engineering, University of Washington, U.S.A.（工程，华盛顿大学，美国）
MechMath: Sorrifier-Driven Formal Decomposition Workflow for Automated Theorem Proving
Academy of Mathematics and Systems Science, CAS（数学与系统科学研究院，中国科学院）；School of Advanced Interdisciplinary Sciences, UCAS（交叉科学学院，中国科学院大学）；School of Mathematical Science, UCAS（数学科学学院，中国科学院大学）
MIRB: Mathematical Information Retrieval Benchmark
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Beijing International Center for Mathematical Research and the New Cornerstone Science Laboratory, Peking University（北京国际数学研究中心和新基石科学实验室，北京大学）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；Center for Intelligent Computing, Great Bay Institute for Advanced Study, Great Bay University（智能计算中心，大湾研究院高级研究所，大湾大学）
