arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.08022cs.AI
收藏

数值敏感性与鲁棒性:探索大型语言模型数学推理中的缺陷

Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Models

Xi’an Jiaotong University(西安交通大学);SGIT AI Lab(SGIT人工智能实验室);A*STAR

Zhishen Sun, Guang Dai, Ivor Tsang, Haishan Ye

AI总结:

针对LLMs数学理解能力的争议,提出含语义无关扰动句子及核心指令缺失的扰动框架,发现其对含数字扰动更敏感、性能降幅大,且可能依赖记忆模板而非逻辑推理,揭示了当前LLMs数学推理的缺陷与局限。

摘要:

大型语言模型(LLMs)在数学推理领域取得了显著进展,但它们是否具备真正的数学理解能力仍存在争议。为了探究这一问题,我们提出了一种新的扰动框架,通过注入额外的语义无关扰动句子并逐渐增加扰动强度,来评估LLMs在复杂环境中的推理能力。同时,我们使用了另一种扰动方法:核心提问指令缺失,以进一步分析LLMs的解题机制。实验结果表明,LLMs在面对不含数字的扰动句子时表现稳定,但也存在鲁棒性边界;随着扰动强度的增加,其性能出现不同程度的下降;而在面对含数字的扰动句子时,性能下降更为显著,大多数较小参数的开源模型下降近10%甚至更多,且随着扰动强度的增强进一步下降,最大降幅达到51.55%。即使是最先进的商业LLMs也出现了3%-10%的性能下降。通过详细分析LLMs的推理过程,我们发现模型对含数值信息的扰动更为敏感,在受到无关数值信息干扰时更易给出错误答案,且扰动强度越高,这些缺陷越明显。同时,在核心提问指令缺失的情况下,模型仍能保持20%-40%的准确率,这表明LLMs可能依赖记忆模板或模式匹配来完成任务,而非逻辑推理。

英文摘要:

LLMs have made significant progress in the field of mathematical reasoning, but whether they have true the mathematical understanding ability is still controversial. To explore this issue, we propose a new perturbation framework to evaluate LLMs' reasoning ability in complex environments by injecting additional semantically irrelevant perturbation sentences and gradually increasing the perturbation intensity. At the same time, we use an additional perturbation method: core questioning instruction missing, to further analyze the LLMs' problem-solving mechanism. The experimental results show that LLMs perform stably when facing perturbation sentences without numbers, but there is also a robustness boundary. As the perturbation intensity increases, the performance exhibits varying degrees of decline; when facing perturbation sentences with numbers, the performance decreases more significantly, most open source models with smaller parameters decrease by nearly or even more than 10%, and further increasing with the enhancement of perturbation intensity, with the maximum decrease reaching 51.55%. Even the most advanced commercial LLMs have seen a 3%-10% performance drop. By analyzing the reasoning process of LLMs in detail, We find that models are more sensitive to perturbations with numerical information and are more likely to give incorrect answers when disturbed by irrelevant numerical information. The higher the perturbation intensity, the more obvious these defects are. At the same time, in the absence of core questioning instruction, models can still maintain an accuracy of 20%-40%, indicating that LLMs may rely on memory templates or pattern matching to complete the task, rather than logical reasoning. In general, our work reveals the shortcomings and limitations of current LLMs in their reasoning capabilities, which is of great significance for the further development of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑