最大似然强化学习
Maximum Likelihood Reinforcement Learning
AI总结：
MaxRL是一种基于采样的强化学习框架，通过最大似然估计提升模型训练效率，实验显示其在多个任务中表现优于现有方法。
摘要：
强化学习是训练在基于采样的设置中使用二元反馈模型的方法，例如导航、代码生成和数学问题解决。在这样的设置中，模型会隐式地诱导一个正确轨迹的似然。然而，我们发现强化学习并不最大化这个似然，而是只优化一个低阶近似。受这一观察的启发，我们引入了最大似然强化学习（MaxRL），一种基于采样的框架，用于使用强化学习技术近似最大似然。MaxRL通过定义一个计算指数族的基于样本的目标函数来解决非可微采样的挑战，这些目标函数在额外采样计算分配时介于标准强化学习和精确最大似然之间。所得到的目标函数允许一个简单且无偏的策略梯度估计器，并在无限计算极限下收敛到最大似然优化。实验上，我们证明MaxRL在所有测试的模型和任务中均优于现有方法，与其GRPO训练的对应物相比，测试时间扩展效率提高了高达20倍。我们还发现MaxRL在额外数据和计算方面具有更好的扩展性。我们的结果表明，MaxRL是一个有前途的框架，用于在基于正确性的设置中扩展RL训练。
英文摘要：
Reinforcement learning (RL) is the method of choice for training models in setups where the objective function can only be evaluated by sampling from the model. Our key observation is that when the feedback is terminal and binary, models implicitly induce a likelihood over correct rollouts. Maximum likelihood would be the natural framework in such settings, but RL is used instead as a workaround to the non-differentiability. We prove that the standard, expected-reward RL formulation is only a first-order approximation of the likelihood. To remedy this mismatch, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a compute-indexed family of sample-based objectives that interpolate between expected-reward RL and maximum likelihood as sampling compute is scaled. The resulting objective is a one-line change to standard RL implementations. MaxRL Pareto-dominates existing methods in all tested models and tasks, achieves up to $20\times$ gains in test-time scaling efficiency over GRPO, and scales more favorably with additional training data and compute.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
