通过反事实推理路径减少信用分配方差
Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
AI总结：
提出反事实比较框架，通过采样多条推理轨迹并利用差异隐式估计过程级优势，将稀疏终端奖励转化为步骤敏感信号，从而改进大语言模型多步推理的信用分配，并引入隐式行为策略优化（IBPO）提升训练稳定性和性能上限。
摘要：
使用大语言模型进行多步推理的强化学习通常依赖于稀疏的终端奖励，这会导致一个条件较差的信用分配问题：最终反馈均匀地传播到所有中间决策。这导致高梯度方差、不稳定的训练和许多无效更新，最终限制了模型的持续改进。我们提出了一种用于信用分配的反事实比较框架。对于每个输入，该框架采样多个推理轨迹，并将它们的差异视为对替代决策的隐式近似。这产生了一个隐式过程级优势估计器，将稀疏终端奖励转化为步骤敏感的学习信号。基于此框架，我们引入了隐式行为策略优化（IBPO），该方法在数学和代码推理基准上显著提高了训练稳定性和性能上限。我们的结果为释放大语言模型的推理潜力指明了一个有前景的方向。
英文摘要：
Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment problem: the final feedback is propagated uniformly across all intermediate decisions. This leads to high gradient variance, unstable training, and many ineffective updates, ultimately limiting sustained model improvement. We propose a counterfactual-comparison framework for credit assignment. For each input, the framework samples multiple reasoning trajectories and treats their differences as implicit approximations to alternative decisions. This yields an implicit process-level advantage estimator that converts sparse terminal rewards into step-sensitive learning signals. Building on this framework, we introduce Implicit Behavior Policy Optimization (IBPO), which substantially improves training stability and the performance ceiling on mathematical and code-reasoning benchmarks. Our results point to a promising direction for unlocking the reasoning potential of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
