推理、代码，还是两者兼有？大型语言模型如何处理数学问题的变化
Reasoning, Code, or Both? How Large Language Models Handle Variations in Math Questions
AI总结：
本研究通过对比链式思维推理、单次代码执行和迭代代码执行三种方法在GSM-Symbolic数据集上的表现，发现代码执行并未提升大型语言模型在数学问题变体上的推理鲁棒性。
摘要：
大型语言模型（LLMs）在数学推理基准测试中取得了令人印象深刻的准确性，但当问题被修改为不同的名字或数字等简单变化时，它们的性能会下降。代码执行方法允许模型生成并运行Python代码，而不是用自然语言进行推理，已被提出作为解决方案，但其对推理鲁棒性（即在问题变体中保持准确性的能力）的影响尚未得到系统测试。本研究在GSM-Symbolic数据集的1000个问题上评估了三种方法：使用链式思维（CoT）提示的纯推理、使用程序辅助语言模型（PAL）的单次代码执行，以及使用逐步编码（SBSC）的迭代代码执行。所有三种方法均在配对的原始问题和修改问题上使用Claude Haiku 4.5运行。CoT是最鲁棒的方法，在扰动下准确率下降1.3个百分点，1.8%的问题被破坏。PAL的鲁棒性最差，准确率下降1.7个百分点，3.1%的问题被破坏，SBSC介于两者之间。尽管这些差异在统计上不显著（$p = .096$），但方向趋势在所有指标上一致，表明无论是单次还是迭代的代码执行，都没有提高小学水平问题变体的推理鲁棒性。
英文摘要：
Large Language Models (LLMs) achieve impressive accuracy on mathematical reasoning benchmarks, yet their performance drops when problems are modified with simple changes like different names or numbers. Code execution methods, which let models generate and run Python code instead of reasoning in natural language, have been proposed as a solution, but their effect on reasoning robustness (the ability to maintain accuracy across problem variations) has not been systematically tested. This study evaluates three approaches on 1,000 problems from the GSM-Symbolic dataset: pure reasoning using chain-of-thought (CoT) prompting, single-shot code execution using Program-Aided Language models (PAL), and iterative code execution using Step-by-Step Coding (SBSC). All three were run on paired original and modified problems using Claude Haiku 4.5. CoT was the most robust method, with an accuracy drop of 1.3 percentage points and 1.8% of problems breaking under perturbation. PAL was the least robust at 1.7 percentage points and 3.1% broke, with SBSC falling in between. Although these differences were not statistically significant ($p = .096$), the directional trend was consistent across all measures, suggesting that code execution, whether single-shot or iterative, does not improve reasoning robustness on grade-school-level problem variations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
