arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.06159cs.LG
收藏

目标策略优化

Target Policy Optimization

Jean Kaddour

AI总结:

TPO通过分离策略更新与目标分布构建,改进了策略梯度方法在稀疏奖励下的性能,适用于表格任务、Transformer序列任务和大模型强化学习。

摘要:

在强化学习中,给定一个提示,我们从模型中采样一组完成项并对其进行评分。随后有两个问题:哪些完成项应获得概率质量,以及如何调整参数以实现这种变化?标准策略梯度方法一次回答这两个问题,因此更新可能会因学习率、裁剪等优化器选择而超调或欠调。我们引入了目标策略优化(TPO),它将这两个问题分开。在评分完成项后,TPO构建一个目标分布$ q_i \propto p_i^{\\,\mathrm{old}} \exp(u_i)$ 并通过交叉熵拟合策略。在采样完成项日志上损失梯度为$p^θ - q$,一旦策略匹配目标,该梯度就消失。在表格任务、变压器序列任务和十亿参数大模型强化学习中,TPO在简单任务中与PG、PPO、GRPO和DG相当,并在稀疏奖励下显著优于它们。代码可在https://github.com/JeanKaddour/tpo获得。

英文摘要:

In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parameters move to realize that change? Standard policy-gradient methods answer both at once, so the update can overshoot or undershoot depending on the learning rate, clipping, and other optimizer choices. We introduce \emph{Target Policy Optimization} (TPO), which separates the two questions. Given scored completions, TPO constructs a target distribution $q_i \propto p_i^{\,\mathrm{old}} \exp(u_i)$ and fits the policy to it by cross-entropy. The loss gradient on sampled-completion logits is $p^θ- q$, which vanishes once the policy matches the target. On tabular bandits, transformer sequence tasks, and billion-parameter LLM RLVR, TPO matches PG, PPO, GRPO, and DG on easy tasks and substantially outperforms them under sparse reward. Code is available at https://github.com/JeanKaddour/tpo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑