脆弱的数感：探究大语言模型中数值推理的极限
A Fragile Number Sense: Probing the Elemental Limits of Numerical Reasoning in LLMs
AI总结：
本研究通过100个复杂度递增的问题测试LLM智能体，发现其在算术、高级运算和质数检验上表现优异，但在24点谜题上失败，表明其数值推理主要依赖模式匹配而非生成性思维。
摘要：
大语言模型（LLMs）已展现出显著的涌现能力，然而其数值推理的鲁棒性仍是一个悬而未决的问题。虽然标准基准通过聚合指标评估LLM在复杂问题集上的推理能力，但这些基准往往掩盖了基础性的弱点。在本工作中，我们通过评估模型在复杂度递增的问题（从基本运算到组合谜题）上的表现，来探究LLM的数学数感。我们测试了多个最先进的基于LLM的智能体，挑战包含100个问题，分为四类：（1）基本算术，（2）高级运算，（3）质数检验，以及（4）24点数字谜题。我们的结果显示，尽管智能体在前三类需要确定性算法执行的问题上取得了高准确率，但它们始终在数字谜题上失败，这突显了其在大规模组合空间上进行启发式搜索的需求是一个显著瓶颈。这些发现表明，智能体的熟练程度主要局限于回忆和执行已知算法，而非进行生成性问题求解。这暗示其表面上的数值推理更类似于复杂的模式匹配，而非灵活的分析性思维，从而限制了它们在需要新颖或创造性数值洞察的任务上的潜力。
英文摘要：
Large Language Models (LLMs) have demonstrated remarkable emergent capabilities, yet the robustness of their numerical reasoning remains an open question. While standard benchmarks evaluate LLM reasoning on complex problem sets using aggregated metrics, they often obscure foundational weaknesses. In this work, we probe LLM mathematical numeracy by evaluating performance on problems of escalating complexity, from constituent operations to combinatorial puzzles. We test several state-of-the-art LLM-based agents on a 100-problem challenge comprising four categories: (1) basic arithmetic, (2) advanced operations, (3) primality checking, and (4) the Game of 24 number puzzle. Our results show that while the agents achieved high accuracy on the first three categories, which require deterministic algorithmic execution, they consistently failed at the number puzzle, underlining its demand for a heuristic search over a large combinatorial space to be a significant bottleneck. These findings reveal that the agents' proficiency is largely confined to recalling and executing known algorithms, rather than performing generative problem-solving. This suggests their apparent numerical reasoning is more akin to sophisticated pattern-matching than flexible, analytical thought, limiting their potential for tasks that require novel or creative numerical insights.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
