训练后强化学习的优先级回放
Prioritized Replay for RL Post-training
AI总结：
提出一种基于经验成功统计的优先级回放方法，用于大型语言模型训练后强化学习，无需预定义难度层级或外部标签，通过动态调整优先级提升学习效率。
摘要：
我们介绍了一种用于大型语言模型训练后强化学习的层级优先级框架。基于深度强化学习中优先级回放的见解，以及先前观察到的在诸如GRPO等方法中，中间成功率的rollouts倾向于产生更强的学习信号，我们的方法根据从经验成功统计中得出的简单模型驱动优先级分数来选择问题。与传统的课程策略强调在训练早期处理更容易的任务不同，所得到的计划自然地将训练集中在既不一致解决也不一致失败的问题上，同时降低那些贡献很少梯度信息的问题的优先级。该方法产生了一个持续适应和自动优先级选择过程，不需要预定义的难度层级、辅助预测器或外部标签。我们进一步引入了轻量级机制以实现实际部署，包括基于堆的优先级采样和定期重新测试已解决和未解决的问题以缓解饥饿和遗忘。总体而言，该方法提供了一种原理上和可扩展的替代方案，用于手动设计的课程，同时使数据选择直接与基于GRPO的训练后动态对齐。
英文摘要：
We introduce a problem-level prioritization framework for RL post-training of large language models. Building on insights from prioritized replay in deep RL, as well as prior observations that rollouts with intermediate success rates tend to produce stronger learning signals under methods such as GRPO, our approach selects problems according to a simple, model-driven priority score derived from empirical success statistics. In contrast to conventional curriculum strategies that emphasize easier tasks early in training, the resulting schedule naturally focuses training on problems that are neither consistently solved nor consistently failed, while deprioritizing those that contribute little gradient information. The method yields a continuously adapting and automatic prioritization process that requires no predefined difficulty tiers, auxiliary predictors, or external labels. We further introduce lightweight mechanisms for practical deployment, including heap-based prioritized sampling and periodic retesting of solved and unsolved problems to mitigate starvation and forgetting. Overall, the approach offers a principled and scalable alternative to manually designed curricula while aligning data selection directly with the dynamics of GRPO-based post-training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
