MSCR：利用多源候选替换探索大语言模型数学推理能力的脆弱性
MSCR: Exploring the Vulnerability of LLMs' Mathematical Reasoning Abilities Using Multi-Source Candidate Replacement
AI总结：
针对LLMs数学推理在轻微输入扰动下鲁棒性缺乏系统研究的问题，提出基于多源候选替换的自动化对抗攻击方法MSCR，结合三种信息源生成并替换语义相似候选词，实验表明其能显著降低模型准确率，凸显LLMs鲁棒性缺陷与效率瓶颈。
摘要：
大语言模型（LLMs）在数学推理等复杂任务中展现出与人类能力相当的性能，但它们在轻微输入扰动下数学推理的鲁棒性仍缺乏系统研究。现有方法普遍存在可扩展性有限、语义保留能力弱和成本高等问题。因此，我们提出MSCR，一种基于多源候选替换的自动化对抗攻击方法。通过结合LLMs嵌入空间中的余弦相似度、WordNet词典以及掩码语言模型的上下文预测这三个信息源，我们为输入问题中的每个单词生成一组语义相似的候选词，随后对这些候选词进行过滤并逐一替换以实施攻击。我们在GSM8K和MATH500基准数据集上对LLMs开展大规模实验，结果显示即使仅涉及单个单词的轻微扰动也能显著降低所有模型的准确率，在GSM8K上最大降幅达49.89%，在MATH500上达35.40%，同时保持扰动问题的高语义一致性。进一步分析表明，扰动不仅导致错误输出，还大幅增加平均响应长度，进而产生更多冗余推理路径和更高的计算资源消耗。这些发现凸显了当前LLMs在数学推理任务中的鲁棒性缺陷和效率瓶颈。
英文摘要：
LLMs demonstrate performance comparable to human abilities in complex tasks such as mathematical reasoning, but their robustness in mathematical reasoning under minor input perturbations still lacks systematic investigation. Existing methods generally suffer from limited scalability, weak semantic preservation, and high costs. Therefore, we propose MSCR, an automated adversarial attack method based on multi-source candidate replacement. By combining three information sources including cosine similarity in the embedding space of LLMs, the WordNet dictionary, and contextual predictions from a masked language model, we generate for each word in the input question a set of semantically similar candidates, which are then filtered and substituted one by one to carry out the attack. We conduct large-scale experiments on LLMs using the GSM8K and MATH500 benchmarks. The results show that even a slight perturbation involving only a single word can significantly reduce the accuracy of all models, with the maximum drop reaching 49.89% on GSM8K and 35.40% on MATH500, while preserving the high semantic consistency of the perturbed questions. Further analysis reveals that perturbations not only lead to incorrect outputs but also substantially increase the average response length, which results in more redundant reasoning paths and higher computational resource consumption. These findings highlight the robustness deficiencies and efficiency bottlenecks of current LLMs in mathematical reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
