发表机构
UC Berkeley; Carnegie Mellon University（加州大学伯克利分校; 卡内基梅隆大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对长视界任务中稀疏奖励导致学习缓慢的问题，本文提出无偏的密集奖励方法渐进式点匹配，在片段级奖励部分进展，理论及实验证明其指数级提升效率，并在困难数学推理任务上取得改进。
AI 中文摘要
当前通过强化学习训练语言模型的范式严重依赖稀疏的结果奖励。然而，当我们追求需要更长、更复杂轨迹的任务时，这种策略会导致学习缓慢。先前的工作试图通过奖励部分进展来解决这个问题；然而，朴素的公式往往有偏，并收敛到次优策略。我们证明，一种简单且无偏的密集奖励公式（我们称之为渐进式点匹配）通过在片段级别奖励部分进展，在理论和经验上（通过合成环境）都能以指数级更高的效率扩展到长视界任务。然后，我们展示了如何利用每个任务单个参考轨迹来实际实例化渐进式点匹配。在极其困难的数学推理问题上，稀疏的结果奖励无法取得任何进展，而片段级奖励在更大的测试时令牌预算下，通过成功率或pass@k衡量时能够实现改进。
英文摘要
Current paradigms for training language models via reinforcement learning rely heavily on sparse outcome rewards. However, as we pursue tasks that require longer and more complicated trajectories, such strategies result in slow learning. Prior work has attempted to address this problem by rewarding partial progress; however, naive formulations are often biased and converge to suboptimal policies. We show that a simple and unbiased dense reward formulation, which we term progressive point matching, scales exponentially more efficiently to long-horizon tasks by rewarding partial progress on a segment level, both theoretically and empirically via synthetic environments. We then show how progressive point matching can be practically instantiated using a single reference trajectory per task. On extremely hard math reasoning problems, sparse outcome rewards cannot make any progress, whereas segment-level rewards enable improvements at larger test-time token budgets when measured by success rate or pass@k.
