RLVR 无需无效样本：面向 LLM 推理的群体优先级离策略优化
RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning
AI总结：
针对强化学习中无效样本导致学习信号不足的问题，提出群体优先级离策略优化（POPO），通过优先级群体重放和解耦重要性采样，在不增加额外采样开销的情况下提升推理性能。
摘要：
基于可验证奖励的强化学习（RLVR）已成为增强大型语言模型（LLMs）推理能力的强大范式。然而，其有效性受到无效训练数据普遍存在的严重阻碍：许多采样提示产生的响应群体要么完全正确，要么完全错误，导致奖励零方差和学习信号有限。最近的先进方法通过大量LLM rollout来过滤无效样本以解决此问题，但代价是相当大的计算开销。替代方法，包括预测性采样和轨迹重放，旨在提高数据效率，但往往仍不充分，并可能引入额外问题，如系统性偏差或次优约束。为解决这些局限性，我们提出了群体优先级离策略优化（POPO），一个简单而有效的框架，无需额外rollout开销即可充分利用有效训练批次。POPO包含两个关键组件：优先级群体重放和解耦离策略优化。前者通过基于近因的重放机制，联合考虑样本质量和离策略程度，用有效的离策略群体替换无效的在策略群体。为进一步缩小离策略差距，POPO采用解耦重要性采样来校正离策略偏差，同时在一致的信任区域约束下保持稳定的策略更新。在包括数学、规划和视觉几何在内的多种推理任务上的实证评估表明，POPO显著加速了RL微调，并在显著减少rollout的情况下实现了强大的推理性能。
英文摘要：
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). However, its effectiveness is substantially hindered by the prevalence of ineffective training data: many sampled prompts yield response groups that are either entirely correct or entirely incorrect, resulting in zero-variance rewards and limited learning signals. Recent state-of-the-art methods address this issue through extensive LLM rollouts to filter ineffective samples, but at the cost of considerable computational overhead. Alternative approaches, including predictive sampling and trajectory replay, aim to improve data efficiency but often remain insufficient and may introduce additional issues such as systematic bias or suboptimal constraints. To address these limitations, we propose Group Prioritized Off-Policy Optimization (POPO), a simple yet effective framework that fully exploits effective training batches without additional rollout overhead. POPO comprises two key components: prioritized group replay and decoupled off-policy optimization. The former replaces ineffective on-policy groups with effective off-policy groups via a recency-based replay mechanism that jointly considers sample quality and the degree of off-policiness. To further mitigate the off-policy gap, POPO employs decoupled importance sampling to correct off-policy bias while maintaining stable policy updates under consistent trust-region constraints. Empirical evaluations across diverse reasoning tasks, including mathematics, planning, and visual geometry, demonstrate that POPO substantially accelerates RL finetuning and achieves strong reasoning performance with significantly fewer rollouts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
