模型改变主意之处：用于可验证奖励强化学习的高效事后分歧定位

中文摘要 AI 辅助

基于分组相对方法的可验证奖励强化学习（RLVR）从轨迹结果的差异中学习。独立采样完整轨迹成本高昂，且未在关键位置显式探索决策空间。对已完成轨迹的反馈可以揭示策略会重新考虑哪些早期选择，从而提示应在何处采样替代延续。我们引入事后分歧定位（HDL），该方法利用事后引起的令牌对数似然变化来选择分支点。HDL生成少量完整根轨迹，并在原始任务上下文下从选定位置用延续填充每个训练组。每个延续复用其根前缀，仅通过新生成的后缀贡献策略更新，从而在减少生成成本的同时，将额外的探索和学习聚焦于分支后的决策。在数学、代码和智能体任务上使用三个模型的实验显示，在轨迹生成效率和任务性能上均有提升。与组大小和训练步数匹配的GRPO相比，HDL在生成令牌上最多减少2.5倍，在轨迹生成的墙钟时间上加速1.8倍。尽管生成预算减少，HDL在三个领域均提升了性能，在智能体任务上最多提升12.5个百分点。

英文摘要

Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5$\times$ reduction in generated tokens and a 1.8$\times$ speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.

发表机构

• University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
• University of California, Berkeley（加州大学伯克利分校）
• ETH Zurich（苏黎世联邦理工学院）
• NVIDIA（英伟达）

机构由 AI 辅助整理，请以论文原文为准。