Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. However, there are increasing debates regarding whether these models truly understand and apply mathematical knowledge or merely rely on shortcuts for mathematical reasoning. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations. We introduce the adversarial grade school math (GSM-Plus) dataset, an extension of GSM8K augmented with various mathematical perturbations. Our experiments on 25 LLMs and 4 prompting techniques show that while LLMs exhibit different levels of math reasoning abilities, their performances are far from robust. In particular, even for problems that have been solved in GSM8K, LLMs can make mistakes when new statements are added or the question targets are altered. We also explore whether more robust performance can be achieved by composing existing prompting methods, in which we try an iterative method that generates and verifies each intermediate thought based on its reasoning goal and calculation result.
Socratic-Zero : Bootstrapping Reasoning via Data-Free Agent Co-evolution
Alibaba Group Holding Limited（阿里巴巴集团控股有限公司）；EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）；Shanghai University of Finance and Economics（上海财经大学）；Wuhan University（武汉大学）；Zhejiang University（浙江大学）
Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions
Idiap Research Institute（Idiap 研究所）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；National Biomarker Centre (NBC), CRUK Manchester Institute（英国癌症研究中心曼彻斯特研究所国家生物标志物中心）
Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs
Integreat - Norwegian Centre for knowledge-driven machine learning（Integreat - 挪威知识驱动机器学习中心）；UiT - The Arctic University of Norway（UiT - 北极大学）；University of Oslo（奥斯陆大学）
Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Team
American International University-Bangladesh（美国国际大学-孟加拉国）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Cornell University（康奈尔大学）；Qatar Computing Research Institute（卡塔尔计算研究所）
