两阶段决策采样假说：理解RL训练LLM中自反思能力的产生
The Two-Stage Decision-Sampling Hypothesis: Understanding the Emergence of Self-Reflection in RL-Trained LLMs
AI总结：
研究探讨了RL训练后LLM中自反思能力的形成机制，提出两阶段决策采样假说，通过梯度归因属性分析发现RL优于SFT的原因在于决策能力的提升而非生成能力。
摘要：
在大型语言模型中，自我反思能力在RL后训练后出现，多轮RL相比SFT有显著优势。然而，统一优化目标如何产生生成解决方案和评估何时修正的能力仍不明确。为此，我们引入梯度归因属性，通过两阶段决策采样假说将策略分解为生成（π_sample）和验证（π_d）两个阶段。证明代理奖励表现出平衡的梯度归因，而SFT和KL惩罚表现出不平衡的梯度归因，长度加权产生不对称正则化，约束π_sample而使π_d未被优化，提供了RL为何优于SFT的理论解释。我们还通过算术推理实验验证理论预测，表明RL的优越泛化能力主要源于决策能力（π_d）的提升，而非生成能力，为思考模型的自我修正提供了基础原理解释。
英文摘要：
Self-reflection capabilities emerge in Large Language Models after RL post-training, with multi-turn RL achieving substantial gains over SFT counterparts. Yet the mechanism of how a unified optimization objective gives rise to functionally distinct capabilities of generating solutions and evaluating when to revise them remains opaque. To address this question, we introduce the Gradient Attribution Property to characterize how reward gradients distribute across policy components, formalized through the Two-Stage Decision-Sampling (DS) Hypothesis, which decomposes the policy into sampling ($π_{sample}$) for generation and decision ($π_{d}$) for verification. We prove that surrogate rewards exhibit Balanced Gradient Attribution, while SFT and KL penalties exhibit Unbalanced Gradient Attribution, with length-weighting creating asymmetric regularization that constrains $π_{sample}$ while leaving $π_{d}$ under-optimized, providing an theoretical explanation of why RL succeeds where SFT fails. We also empirically validate our theoretical predictions on arithmetic reasoning demonstrates that RL's superior generalization stems primarily from improved decision-making ($π_{d}$) rather than sampling capabilities, providing a first-principles mechanistic explanation for self-correction in thinking models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
