思维的幻觉：通过问题复杂性的视角理解推理模型的优势与局限
The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
AI总结：
通过问题复杂性分析，研究揭示大规模推理模型在不同复杂度任务中的表现差异及局限性。
摘要：
近年来，语言模型的最新一代引入了大规模推理模型（LRMs），这些模型在提供答案前会生成详细的思考过程。尽管这些模型在推理基准测试中表现出改进的性能，但其基本能力、扩展性质和局限性仍不够理解。当前的评估主要集中在已建立的数学和编码基准上，强调最终答案的准确性。然而，这种评估范式往往受到污染，且不提供关于推理轨迹的见解。在本工作中，我们通过可控的谜题环境系统地调查这些差距，该环境允许精确地操纵复杂性，同时保持一致的逻辑结构。这种设置使我们能够分析不仅最终答案，还有内部推理轨迹，从而深入了解LRMs的思考方式。通过广泛的实验，我们显示LRMs在某些复杂性之外会出现完全的准确性崩溃。此外，它们表现出一种反直觉的扩展限制：推理努力随着问题复杂性增加到一定程度后，即使有剩余的token预算，也会下降。通过在相同推理计算下将LRMs与标准LLM模型进行比较，我们识别出三种性能区域：（1）低复杂度任务中标准模型优于LRMs，（2）中等复杂度任务中LRMs表现出优势，（3）高复杂度任务中两种模型都面临完全崩溃。我们发现LRMs在精确计算上有局限：它们无法使用显式算法，并且在不同尺度上推理不一致。我们还更深入地研究了推理轨迹，研究了探索的解决方案模式，并分析了模型的计算行为，揭示了它们的优势、局限以及关于其推理能力的疑问。
英文摘要：
Recent generations of language models have introduced Large Reasoning Models (LRMs) that generate detailed thinking processes before providing answers. While these models demonstrate improved performance on reasoning benchmarks, their fundamental capabilities, scaling properties, and limitations remain insufficiently understood. Current evaluations primarily focus on established math and coding benchmarks, emphasizing final answer accuracy. However, this evaluation paradigm often suffers from contamination and does not provide insights into the reasoning traces. In this work, we systematically investigate these gaps with the help of controllable puzzle environments that allow precise manipulation of complexity while maintaining consistent logical structures. This setup enables the analysis of not only final answers but also the internal reasoning traces, offering insights into how LRMs think. Through extensive experiments, we show that LRMs face a complete accuracy collapse beyond certain complexities. Moreover, they exhibit a counterintuitive scaling limit: their reasoning effort increases with problem complexity up to a point, then declines despite having remaining token budget. By comparing LRMs with their standard LLM counterparts under same inference compute, we identify three performance regimes: (1) low-complexity tasks where standard models outperform LRMs, (2) medium-complexity tasks where LRMs demonstrates advantage, and (3) high-complexity tasks where both models face complete collapse. We found that LRMs have limitations in exact computation: they fail to use explicit algorithms and reason inconsistently across scales. We also investigate the reasoning traces in more depth, studying the patterns of explored solutions and analyzing the models' computational behavior, shedding light on their strengths, limitations, and raising questions about their reasoning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
