LLMs have made significant progress in the field of mathematical reasoning, but whether they have true the mathematical understanding ability is still controversial. To explore this issue, we propose a new perturbation framework to evaluate LLMs' reasoning ability in complex environments by injecting additional semantically irrelevant perturbation sentences and gradually increasing the perturbation intensity. At the same time, we use an additional perturbation method: core questioning instruction missing, to further analyze the LLMs' problem-solving mechanism. The experimental results show that LLMs perform stably when facing perturbation sentences without numbers, but there is also a robustness boundary. As the perturbation intensity increases, the performance exhibits varying degrees of decline; when facing perturbation sentences with numbers, the performance decreases more significantly, most open source models with smaller parameters decrease by nearly or even more than 10%, and further increasing with the enhancement of perturbation intensity, with the maximum decrease reaching 51.55%. Even the most advanced commercial LLMs have seen a 3%-10% performance drop. By analyzing the reasoning process of LLMs in detail, We find that models are more sensitive to perturbations with numerical information and are more likely to give incorrect answers when disturbed by irrelevant numerical information. The higher the perturbation intensity, the more obvious these defects are. At the same time, in the absence of core questioning instruction, models can still maintain an accuracy of 20%-40%, indicating that LLMs may rely on memory templates or pattern matching to complete the task, rather than logical reasoning. In general, our work reveals the shortcomings and limitations of current LLMs in their reasoning capabilities, which is of great significance for the further development of LLMs.
The Hong Kong Polytechnic University(香港理工大学);Southern University of Science and Technology(南方科技大学);Tsinghua University(清华大学);Reallm Labs(Reallm实验室);Wuhan University(武汉大学);The University of Hong Kong(香港大学)
Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学上海智能信息处理重点实验室)
RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning
Artificial Intelligence Thrust, Information Hub, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China(人工智能 thrust,信息中心,香港科学与技术大学(广州),广州,中国)
Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning
The Hong Kong Polytechnic University(香港理工大学);University of California, Santa Barbara(加州大学圣芭芭拉分校);Peking University(北京大学);The Hong Kong University of Science and Technology(香港科学与技术大学);The University of Hong Kong(香港大学)
Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics
Pennsylvania State University(宾夕法尼亚州立大学);City University of New york(纽约城市大学);New York University(纽约大学);Amirkabir University of Technology(阿米尔卡比尔理工大学);Bocconi University(博科尼大学);Autodesk(欧特克公司)
Exploring Communication Strategies for Collaborative LLM Agents in Mathematical Problem-Solving
Institute for Intelligent Systems, University of Memphis, Memphis, TN, USA(智能系统研究所,密苏里大学);Department of Electrical and Computer Engineering, University of Memphis, Memphis, TN, USA(电气与计算机工程系,密苏里大学);AI4STEM Education Center, University of Georgia, Athens, GA, USA(AI4STEM教育中心,佐治亚大学);National GENIUS Center, Athens, GA, USA(国家GENIUS中心);Department of Mathematics, Science, and Social Studies Education, University of Georgia, Athens, GA, USA(数学、科学与社会科学教育系,佐治亚大学);Faculty of Education, The University of Hong Kong, Hong Kong, PR China(教育学院,香港大学);Educational Testing Service, Princeton, NJ, USA(教育测评服务);Department of Applied Social Sciences, Hong Kong Polytechnic University, Hong Kong, PR China(应用社会科学系,香港理工大学);Department of Psychology, University of Memphis, Memphis, TN, USA(心理学系,密苏里大学)
该研究使用 GPT-4o 在 MATH 数据集上比较双智能体数学解题的四种沟通模式,发现同伴协作准确率最高,并表明有效对话行为是提升 AI 教育协作解题的关键。
A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Data
AnnLab(安澜实验室);Institute of Semiconductors, Chinese Academy of Sciences(中国科学院半导体研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Key Laboratory of Semiconductor Neural Network Intelligent Sensing and Computing Technology(北京市半导体神经网络智能感知与计算重点实验室);Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences(中国科学院大学材料科学与光电技术学院)
CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions
University of Massachusetts Lowell(马萨诸塞大学洛文分校);San Jose State University(圣何塞州立大学);Massachusetts Institute of Technology(麻省理工学院);Dartmouth College(达特茅斯学院);Amazon AGI(亚马逊人工智能研究院)
To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Huawei International, Singapore(华为国际(新加坡));State University of New York at Buffalo(纽约州立大学布法罗分校)