CoT-Space: 一种通过强化学习实现内部慢思考的理论框架
Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought
AI总结：
本文提出CoT-Space理论框架，通过强化学习将推理过程从离散的token预测任务转化为连续的推理层面语义空间中的优化过程，揭示了测试时扩展中最优CoT长度的收敛是欠拟合与过拟合基本权衡的自然结果。
摘要：
测试时扩展，主要通过强化学习（RL）中的多步链式推理（CoT）体现，已成为增强大型语言模型（LLMs）推理能力的关键范式。然而，仍存在显著的理论空白：传统token级分析无法捕捉推理层面扩展的宏观动态。为此，我们引入CoT-Space，一种新的理论框架，将推理过程从离散的token预测任务转换为连续的推理层面语义空间中的优化过程。通过从噪声和风险视角建模推理轨迹，并复兴经典学习理论中的基础原理，我们证明观察到的收敛到最优CoT长度是欠拟合与过拟合基本权衡的自然结果。我们进一步利用RL作为工具，在实验中激发并验证这些结果。我们的发现为通过RL实现内部测试时扩展提供了机制解释，为现代LLMs中优化推理轨迹提供了系统性的理论基础。
英文摘要：
Test-time scaling, primarily manifested through multi-step Chain-of-Thought (CoT) reasoning via Reinforcement Learning (RL), has emerged as a pivotal paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). However, a significant theoretical gap persists: traditional token-level analysis fails to capture the macroscopic dynamics of reasoning-level scaling. To address this, we introduce CoT-Space, a novel theoretical framework that recasts the reasoning process from a discrete token-prediction task to an optimization process within a continuous, reasoning-level semantic space. By modeling the reasoning trajectory from both noise and risk perspectives and revitalizing foundational principles from classical learning theory, we demonstrate that the observed convergence to an optimal CoT length is a natural consequence of the fundamental trade-off between underfitting and overfitting. We further utilize RL as a tool to elicit and verify these results in our experiments. Our findings provide a mechanistic explanation for the internal test-time scaling via RL, offering a principled theoretical foundation to optimize reasoning trajectories in modern LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
