发表机构
University of Wisconsin–Madison; University of Washington（威斯康星大学麦迪逊分校; 华盛顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对RLVR中饱和组导致的rollout浪费问题，提出SARA方法，通过序贯分配规则减少rollout用量，在1.5B/3B模型上实现高效计算的同时保持性能。
AI 中文摘要
带可验证奖励的强化学习（RLVR）受rollout生成瓶颈制约，然而许多采样提示会产生饱和组（所有回答全对或全错），其零奖励方差无法提供策略梯度信号。现有补救措施要么过采样更大候选池并丢弃饱和提示（动态采样），付出大量额外rollout成本；要么在采样前预测提示难度，这在策略变化时鲁棒性差。我们发现，一组的有效性通常在其前几次rollout内就已确定，将完整组用于已判定的提示会造成浪费。我们将每一步rollout收集建模为预算约束下的序贯分配（最优停止）问题，并提出SARA（Sequential Adaptive Rollout Allocation）。SARA为每个提示的成功率维护Beta后验，评估组有效性的闭式预测器，采用双阈值、SPRT风格规则：提交有效组，经短暂探测后放弃饱和组，将释放的预算重新分配给新提示，无需任何额外预测rollout。我们证明了放弃的可靠性、预期rollout节省、固定预算下的收益优势，以及有效组收益与GRPO梯度范数的关联。在单GPU上用1.5B/3B模型进行数学推理与规划任务中，SARA的性能与DPS相当（均低于DS oracle），但比DS少用22%的rollout；将SARA与DPS结合，可获得略高于DS的最佳准确率，同时减少67%的rollout（成本接近均匀）。
英文摘要
Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).
