arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.05802cs.LG
收藏

选择性回放:多样本代理强化学习中的中程轨迹终止

Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL

Fudan University(复旦大学)

Zhiyuan Zhai, Xin Wang

AI总结:

本文提出一种中程轨迹终止方法,通过在轨迹中途检测行动序列的编辑距离来提前停止无方差组,从而减少计算开销并提升测试任务的成功率。

摘要:

组相对强化学习训练(GRPO)为每个训练提示采样一个小的并行轨迹组,并利用组内奖励分布来计算每条轨迹的优势。在代理环境中,每个轨迹是一个长的多轮对话,每一步调用一个大语言模型,因此这种多样本乘数主导了总训练成本。当每个提示的轨迹以相同的奖励结束时,组内奖励方差为零且不贡献梯度,因此额外的轨迹没有信息;此类组在实践中很常见(通常占所有组的40%),因此浪费的计算量相当大而非边际。现有方法在提示级别过滤此类组,要么在支付轨迹后或在任何轨迹开始前,但两者均未利用在轨迹本身期间可用的信息。我们相反地问,在中间步骤的轨迹部分之间组内分歧是否能预测组将产生零方差:当并行轨迹已经收敛到相同的行动前缀时,组将产生单一奖励,我们可以提前停止。我们提出一个一参数门,当组内部分行动序列的平均成对前缀编辑距离低于阈值时停止该组。在ALFWorld上60次迭代的on-policy GRPO运行中,平均四随机种子,加权门臂在墙钟时间上完成速度提高了10.7%(bootstrap 95% CI排除0),并在50个未见任务上的保持成功率提高了+2.5个百分点,该保持增益可追溯到可测量的零优势梯度批次稀释减少。代码可在https://github.com/zhiyuanZhai20/selective-rollout获得。

英文摘要:

Group-relative RL training (GRPO) samples a small group of parallel rollouts for every training prompt and uses their within-group reward spread to compute per-trajectory advantages. In agentic environments each rollout is a long multi-turn dialogue with one LLM call per step, so this multi-sample multiplier dominates the total training cost. When every rollout of a prompt ends with the same reward, the group has zero reward variance and contributes no gradient, so the extra rollouts add no information; such groups are common in practice (typically around 40% of all groups), so the wasted-compute fraction is substantial rather than marginal. Existing methods filter such groups at the prompt level, either after their rollouts are paid for or before any rollout begins, but both decide without using information that becomes available during the rollout itself. We instead ask whether the in-group divergence between the partial trajectories at an intermediate step can already predict that the group will be zero-variance: when the parallel rollouts have already converged on the same action prefix, the group is on track to produce a single reward, and we can stop early. We propose a one-parameter gate that stops a group when the mean pairwise prefix edit distance between its partial action sequences falls below a threshold. On a 60-iteration on-policy GRPO run on ALFWorld with Qwen2.5-7B, averaged over four random seeds, the gated arm finishes 10.7% faster in wall-clock (bootstrap 95% CI excludes 0) and shifts held-out success rate on 50 unseen tasks by +2.5 pp, with the held-out gain tracing to a measurable reduction in zero-advantage gradient-batch dilution. Code is available at https://github.com/zhiyuanZhai20/selective-rollout.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑