A well-defined benchmark is essential for measuring and accelerating research progress of machine learning models. In this paper, we present a benchmark for high-level mathematical reasoning and study the reasoning capabilities of neural sequence-to-sequence models. We build a non-synthetic dataset from the largest repository of proofs written by human experts in a theorem prover. The dataset has a broad coverage of undergraduate and research-level mathematical and computer science theorems. In our defined task, a model is required to fill in a missing intermediate proposition given surrounding proofs. This task provides a starting point for the long-term goal of having machines generate human-readable proofs automatically. Our experiments and analysis reveal that while the task is challenging, neural models can capture non-trivial mathematical reasoning. We further design a hierarchical transformer that outperforms the transformer baseline.
An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Department of Computer Science University of Illinois Urbana–Champaign（计算机科学系伊利诺伊大学厄巴纳-香槟分校）；Department of Computer Science Stanford University（计算机科学系斯坦福大学）
Lila: A Unified Benchmark for Mathematical Reasoning
Arizona State Univeristy（亚利桑那州立大学）；The Allen Institute for AI（艾伦人工智能研究所）；UCLA（加州大学洛杉矶分校）；Harvard University（哈佛大学）；Georgia Institute of Technology（佐治亚理工学院）
DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometry
East China Normal University（华东师范大学）；Zhongguancun Academy（中关村学院）；Huazhong University of Science and Technology（华中科技大学）；Peking University（北京大学）；Zhengzhou University（郑州大学）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）
The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models
Texas A&M University（德克萨斯A&M大学）；Harvard University（哈佛大学）；Vanderbilt University（范德堡大学）；Stanford University（斯坦福大学）；DARPA（美国国防高级研究计划局）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Multimodal Mathematical Reasoning with Diverse Solving Perspective
Tongji University（同济大学）；National University of Singapore（新加坡国立大学）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；Tencent（腾讯）；Center for Future Media, University of Electronic Science and Technology of China（电子科技大学未来媒体中心）
RoMath: A Mathematical Reasoning Benchmark in Romanian
POLITEHNICA Bucharest National University of Science and Technology（巴格达理工大学科学与技术国立大学）；Interdisciplinary School of Doctoral Studies, University of Bucharest（布加勒斯特大学跨学科博士研究学院）
Multimodal Mathematical Reasoning Embedded in Aerial Vehicle Imagery: Benchmarking, Analysis, and Exploration
Department of Physics, J.K. Institute of Science（J.K.科学研究院物理系）；World Scientific University（世界科学大学）；University of Intelligent Studies（智能研究大学）；East China Normal University（华东师范大学）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology（哈尔滨工业大学）
