Large language models (LLMs) have achieved strong performance on reasoning benchmarks, yet their ability to solve real-world problems requiring end-to-end workflows remains unclear. Mathematical modeling competitions provide a stringent testbed for evaluating such end-to-end problem-solving capability. We propose a problem-oriented, stage-wise evaluation framework that assesses LLM performance across modeling stages using expert-verified criteria. We validate the framework's reliability by comparing automatic scores with independent human expert judgments on problems from the China Postgraduate Mathematical Contest in Modeling, demonstrating substantially stronger alignment than existing evaluation schemes. Using this framework, we reveal a comprehension-execution gap in state-of-the-art LLMs: while they perform well in early stages such as problem identification and formulation, they exhibit persistent deficiencies in execution-oriented stages including model solving, code implementation, and result analysis. These gaps persist even with increased model scale. We further trace these failures to insufficient specification, missing verification, and lack of validation, with errors propagating across stages without correction. Our findings suggest that bridging this gap requires approaches beyond model scaling, offering insights for applying LLMs to complex real-world problem solving.
Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖先进研究院技术研究所）；Huawei Noah’s Ark Lab（华为诺亚实验室）
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
Shanghai AI Laboratory（上海人工智能实验室）；MMLab, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Shanghai Jiao Tong University（上海交通大学）；Great Bay University（大湾区大学）
Lean Workbook: A large-scale Lean problem set formalized from natural language math problems
Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）
Bolzano: From Expert-Guided Proof Search to Automated Open-Problem Solving
Computer Science Institute, Charles University（查尔斯大学计算机科学研究所）；Institute of Formal and Applied Linguistics, Charles University（查尔斯大学形式与应用语言学研究所）；Department of Applied Mathematics, Charles University（查尔斯大学应用数学系）；Institute of Mathematics, Czech Academy of Sciences（捷克科学院数学研究所）
FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs
Columbia University（哥伦比亚大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Rensselaer Polytechnic Institute（拉特格斯理工学院）；University of Montreal（蒙特利尔大学）；McGill University（麦吉尔大学）；MBZUAI（麦吉尔大学人工智能研究所）；Cardiff University（卡迪夫大学）；The University of Manchester（曼彻斯特大学）；Harvard University（哈佛大学）
