语言模型推理中带有重置的信用分配
Credit Assignment with Resets in Language Model Reasoning
AI总结：
提出随机重置策略优化（RRPO）和自重置策略优化（SRPO）两种方法，通过重置到中间状态并重新采样反事实延续来改进语言模型多步推理中的信用分配，SRPO在多个推理基准上优于标准GRPO和RRPO。
摘要：
当代使用可验证奖励方法的强化学习通过对轨迹中的所有令牌统一分配单一结果奖励来对多步推理进行语言模型后训练。这种统一分配忽略了哪些步骤促成了成功或失败。改进信用分配可以通过实现对错误推理步骤的针对性细化来解决这一限制，而不是统一更新整个轨迹。重置是一种简单的机制，通过返回到中间状态并重新采样反事实延续来实现更精确的信用分配，从而将结果差异归因于该点做出的决策。我们提出了两种这样的方法：随机重置策略优化（RRPO），其中重置状态从推理步骤中随机抽取；以及自重置策略优化（SRPO），其中模型自我定位错误轨迹中的错误步骤并在此重置。我们在保守策略迭代（CPI）框架内分析了这些方法。通过针对可改进状态的信用分配预言机扩展CPI，相比于随机重置可证明改进。在多个模型和推理基准上，SRPO通过仅在自我定位的重置处采样多个后缀延续并从其奖励中学习，仅使用模型本身且无需外部监督，始终优于标准GRPO和RRPO。
英文摘要：
Contemporary reinforcement learning with verifiable reward methods post-train language models on multi-step reasoning by assigning a single outcome reward uniformly across all tokens in a trajectory. Such uniform assignment ignores which steps contributed to success or failure. Improving credit assignment can address this limitation by enabling targeted refinement of faulty reasoning steps, rather than updating entire trajectories uniformly. Resets are one such simple mechanism, enabling more precise credit assignment by returning to an intermediate state and resampling counterfactual continuations, so that outcome differences can be attributed to decisions made at that point. We propose two such methods: Random-Reset Policy Optimization (RRPO), where reset states are drawn randomly from reasoning steps, and Self-Reset Policy Optimization (SRPO), where the model self-localizes the erroneous step in an incorrect trajectory and resets there. We analyze these methods within the Conservative Policy Iteration (CPI) framework. Extending CPI with a credit-assignment oracle that targets improvable states yields provable improvements over random resets. Across models and reasoning benchmarks, SRPO consistently outperforms standard GRPO and RRPO by sampling multiple suffix continuations at a self-localized reset and learning from their rewards, using only the model itself with no external supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
