Group-relative RL training (GRPO) samples a small group of parallel rollouts for every training prompt and uses their within-group reward spread to compute per-trajectory advantages. In agentic environments each rollout is a long multi-turn dialogue with one LLM call per step, so this multi-sample multiplier dominates the total training cost. When every rollout of a prompt ends with the same reward, the group has zero reward variance and contributes no gradient, so the extra rollouts add no information; such groups are common in practice (typically around 40% of all groups), so the wasted-compute fraction is substantial rather than marginal. Existing methods filter such groups at the prompt level, either after their rollouts are paid for or before any rollout begins, but both decide without using information that becomes available during the rollout itself. We instead ask whether the in-group divergence between the partial trajectories at an intermediate step can already predict that the group will be zero-variance: when the parallel rollouts have already converged on the same action prefix, the group is on track to produce a single reward, and we can stop early. We propose a one-parameter gate that stops a group when the mean pairwise prefix edit distance between its partial action sequences falls below a threshold. On a 60-iteration on-policy GRPO run on ALFWorld with Qwen2.5-7B, averaged over four random seeds, the gated arm finishes 10.7% faster in wall-clock (bootstrap 95% CI excludes 0) and shifts held-out success rate on 50 unseen tasks by +2.5 pp, with the held-out gain tracing to a measurable reduction in zero-advantage gradient-batch dilution. Code is available at https://github.com/zhiyuanZhai20/selective-rollout.
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Carnegie Mellon University(卡内基梅隆大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Meta AI
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算与智能学院);Beijing Zhongguancun Academy(北京中关村学院);XinzhuAI(芯智科技)
SkillScaffold: Scaffolding Agentic Reinforcement Learning with Policy-Adaptive Skill Credit Assignment and Selective Injection
Zhejiang University(浙江大学);Kuaishou Technology(快手科技);University of the Chinese Academy of Sciences(中国科学院大学);Zhejiang University of Science and Technology(浙江科技学院)
What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Nanjing University(南京大学)
Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
Beijing Institute of Technology(北京理工大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);DataCanvas;Beijing University of Posts and Telecommunications(北京邮电大学);Shenzhen MSU-BIT University(深圳MSU-BIT大学)
STEPS: Selective On-Policy Self-Distillation for Reasoning
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院);AMAP, Alibaba Group(阿里集团AMAP);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Tsinghua University(清华大学)
ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
Alibaba Group(阿里巴巴集团);Beijing Institute of Technology(北京理工大学);Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Zhongguancun Academy(中关村学院)