基于轨迹级别优势优先经验回放的GRPO
Rollout-Level Advantage-Prioritized Experience Replay for GRPO
AI总结：
针对GRPO样本效率低的问题，提出轨迹级经验回放缓冲器，通过年龄驱逐限制陈旧性、新鲜锚定组合保持在线策略、按优势幅度优先采样，在多个数学基准上显著提升性能。
摘要：
基于可验证奖励的GRPO强化学习是后训练推理LLM的标准方法，但样本效率低下。每个轨迹仅用于一次梯度更新后被丢弃。朴素回放在此设置中不适用，因为LLM策略每步梯度变化快，存储的轨迹会变得陈旧并破坏训练稳定性。我们提出一种面向GRPO的轨迹级回放缓冲器，存储和采样单个轨迹而非整组。缓冲器通过年龄驱逐限制陈旧性：任何超过tau_max训练步数的轨迹被移除。缓冲器还通过新鲜锚定组合保留在线策略数据：每个批次保留其新鲜的在线策略轨迹，并拼接从缓冲器中单独抽取的回放轨迹。我们按每个轨迹的优势幅度进行优先回放，并回收优势大的单个轨迹。在三个Qwen3-Base规模、五个数学基准上，我们的方法优于GRPO和朴素回放基线。所有规模均获得正向增益，且随模型增大而增长。最大增益在4B规模上，五个基准平均提升+4.35个百分点。在联合衡量准确率和token效率的AES指标下，与GRPO的效率差距同样在4B最大，为+0.579。
英文摘要：
Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs. It remains sample inefficient. Each rollout is used for a single gradient update and then discarded. Naive replay is not well suited in this setting because LLM policies drift quickly per gradient step. Stored rollouts therefore become stale and can destabilize training. We propose a rollout-level replay buffer for GRPO that stores and samples individual rollouts rather than whole groups. The buffer bounds staleness through age eviction. Any rollout older than tau_max training steps is removed. The buffer also preserves on-policy data via fresh-anchored composition. Each batch keeps its fresh on-policy rollouts and then concatenates replay rollouts drawn separately from the buffer. We prioritize replay by per-rollout advantage magnitude and recycle individual rollouts whose advantages are large. Across three Qwen3-Base scales on five math benchmarks, our method outperforms GRPO and naive replay baselines. Gains are positive at every scale and reach +1.66 pp on the five-benchmark average at 4B. Under an AES metric that jointly measures accuracy and token efficiency, our method is the only condition with a positive margin over GRPO at every scale.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
