持续扩展的幻觉回报：衡量LLM的长周期执行
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
AI总结：
本文通过分析LLM在长周期任务中的执行能力，揭示了持续扩展LLM并非必然导致回报递减，而是执行能力的提升能显著改善长周期任务表现。
摘要：
持续扩展大型语言模型（LLM）是否会产生边际收益递减？本文表明，短期任务基准可能产生执行进步减缓的错觉，因为即使单步准确性的小幅提升，也能累积成任务长度的指数级改进。我们进一步认为，LLM在简单任务延长时的失败源于执行错误而非推理能力不足。因此，我们提出通过显式提供解决长周期任务所需的知识和计划来隔离执行能力。首先，我们发现更大模型在小模型几乎完美单步准确性的情况下，能够正确执行更多步骤。然后我们观察到，随着步骤数的增加，模型的每步准确性会下降。这不仅仅是因为长上下文限制——有趣的是，我们观察到自我条件效应：当上下文包含先前步骤的错误时，模型更可能犯错。自我条件效应不只通过扩大模型规模就能减少。但，我们发现思考可以缓解自我条件效应，并且使模型能够在单步中执行更长的任务。最后，我们通过基准测试前沿思考模型在单步中能执行的任务长度。总体而言，通过关注执行能力，我们希望解决LLM在复杂推理问题上表现良好但在简单任务延长时失败的争论，并强调扩大模型规模和顺序测试时间计算对长周期任务的巨大好处。
英文摘要：
Does continued scaling of large language models (LLMs) yield diminishing returns? In this work, we show that short-task benchmarks may give an illusion of slowing progress, as even marginal gains in single-step accuracy can compound into exponential improvements in the length of tasks a model can successfully complete. Then, we argue that failures of LLMs when simple tasks are made longer arise from mistakes in execution, rather than an inability to reason. So, we propose isolating execution capability, by explicitly providing the knowledge and plan needed to solve a long-horizon task. First, we find that larger models can correctly execute significantly more turns even when small models have near-perfect single-turn accuracy. We then observe that the per-step accuracy of models degrades as the number of steps increases. This is not just due to long-context limitations -- curiously, we observe a self-conditioning effect -- models become more likely to make mistakes when the context contains their errors from prior turns. Self-conditioning does not reduce by just scaling the model size. But, we find that thinking mitigates self-conditioning, and also enables execution of much longer tasks in a single turn. We conclude by benchmarking frontier thinking models on the length of tasks they can execute in a single turn. Overall, by focusing on the ability to execute, we hope to reconcile debates on how LLMs can solve complex reasoning problems yet fail at simple tasks when made longer, and highlight the massive benefits of scaling model size and sequential test-time compute for long-horizon tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
