Jackpot: 为极端演员-策略不匹配强化学习的最优预算拒绝采样
Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learning
AI总结：
Jackpot通过最优预算拒绝采样方法，有效减少rollout模型与策略之间的分布差异，提升大语言模型强化学习的训练稳定性与效率。
摘要：
强化学习（RL）用于大语言模型（LLMs）仍然昂贵，尤其是因为rollout过程昂贵。将rollout生成与策略优化（例如利用更高效的模型进行rollout）解耦可以带来显著的效率提升，但这样做会引入严重的分布不匹配，从而破坏学习。我们提出了Jackpot，一个框架，利用最优预算拒绝采样（OBRS）直接减少rollout模型与演进策略之间的差异。Jackpot集成了一个原则性的OBRS过程，一个联合更新策略和rollout模型的统一训练目标，以及通过top-k概率估计和批量级偏差校正实现的高效系统实现。我们的理论分析表明，OBRS在可控的接受预算下能够持续将rollout分布推向目标分布。实验表明，与重要性采样基线相比，\sys显著提高了训练稳定性，当训练Qwen3-8B-Base进行最多300次更新步骤（批量大小64）时，性能可与on-policy RL相媲美。综上，我们的结果表明，基于OBRS的对齐使我们更接近于实际有效的rollout生成与策略优化解耦，用于LLMs的RL。
英文摘要：
Reinforcement learning (RL) for large language models (LLMs) remains expensive, particularly because the rollout is expensive. Decoupling rollout generation from policy optimization (e.g., leveraging a more efficient model to rollout) could enable substantial efficiency gains, yet doing so introduces a severe distribution mismatch that destabilizes learning. We propose Jackpot, a framework that leverages Optimal Budget Rejection Sampling (OBRS) to directly reduce the discrepancy between the rollout model and the evolving policy. Jackpot integrates a principled OBRS procedure, a unified training objective that jointly updates the policy and rollout models, and an efficient system implementation enabled by top-$k$ probability estimation and batch-level bias correction. Our theoretical analysis shows that OBRS consistently moves the rollout distribution closer to the target distribution under a controllable acceptance budget. Empirically, \sys substantially improves training stability compared to importance-sampling baselines, achieving performance comparable to on-policy RL when training Qwen3-8B-Base for up to 300 update steps of batchsize 64. Taken together, our results show that OBRS-based alignment brings us a step closer to practical and effective decoupling of rollout generation from policy optimization for RL for LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
