arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.24840cs.CL
收藏

在生成过程中剪枝:用于更快更好RLVR的在线回滚剪枝

Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR

University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Amazon(亚马逊)

Haobo Xu, Sirui Chen, Ruizhong Qiu, Yuchen Yan, Chen Luo, Monica Cheng, Jingrui He, Hanghang Tong

AI总结:

本文提出arrol方法,通过在生成过程中剪枝回滚,提高RLVR的训练效率和准确性,实验显示在Qwen-3和LLaMA-3.2模型上平均准确率提升2.30至2.99,训练速度提升1.7倍,测试时扩展性提升8.33。

摘要:

可验证奖励的强化学习(RLVR)显著提升了大语言模型(LLMs)的推理能力。然而,GRPO和DAPO等方法因每次提示都需要采样大量回滚而产生显著的计算成本。此外,在RLVR中,相对优势往往稀疏:许多样本几乎全部正确或全部错误,导致组内奖励方差低,学习信号弱。本文引入arrol(通过在线回滚剪枝加速RLVR),一种在生成过程中剪枝回滚的方法,通过显式引导幸存的回滚更加平衡正确性以增强学习信号。具体而言,arrol在生成过程中实时训练一个轻量级的质量头,预测部分回滚的成功概率,并利用它进行早期剪枝决策。所学的质量头还能进一步在测试时扩展中加权候选者以提高推理准确性。为提高效率,我们提出了一种系统设计,在推理引擎内剪枝回滚,并重新分批剩余的回滚以进行对数概率计算和策略更新。在Qwen-3和LLaMA-3.2模型(1B-8B)上,arrol在平均准确率上提升2.30至2.99,实现高达1.7倍的训练加速,并在测试时扩展中产生高达8.33的额外增益。代码可在https://github.com/Hsu1023/ARRoL获取。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, methods such as GRPO and DAPO suffer from substantial computational cost, since they rely on sampling many rollouts for each prompt. Moreover, in RLVR the relative advantage is often sparse: many samples become nearly all-correct or all-incorrect, yielding low within-group reward variance and thus weak learning signals. In this paper, we introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method that prunes rollouts during generation while explicitly steering the surviving ones more correctness-balanced to enhance learning signals. Specifically, arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions. The learned quality head can further weigh candidates to improve inference accuracy during test-time scaling. To improve efficiency, we present a system design that prunes rollouts inside the inference engine and re-batches the remaining ones for log-probability computation and policy updates. Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models (1B-8B), arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup, and yielding up to +8.33 additional gains in average accuracy in test-time scaling. The code is available at https://github.com/Hsu1023/ARRoL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑