Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.
Rollout-Level Advantage-Prioritized Experience Replay for GRPO
Department of Electrical and Computer Engineering, Seoul National University(首尔国立大学电子与计算机工程系);Interdisciplinary Program in AI, Seoul National University(首尔国立大学人工智能跨学科项目);AIIS, ASRI, INMC, and ISRC, Seoul National University(首尔国立大学人工智能研究所、人工智能研究机构、智能网络与计算中心及人工智能科学研究中心)
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算与智能学院);Beijing Zhongguancun Academy(北京中关村学院);XinzhuAI(芯智科技)
How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVR
TCS Research Department of CSE(TCS计算机科学系研究部);IIT Madras(印度理工学院马德拉斯分校);Department of Data Science & AI(数据科学与人工智能系);Wadhwani School of Data Science & AI(Wadhwani数据科学与人工智能学院)
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
South China University of Technology(华南理工大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学);Huawei 2012 Laboratories(华为2012实验室);Shenzhen Technology University(深圳技术大学)
Learning with a Single Rollout via Pass@k Policy Optimization
University of Alberta(阿尔伯塔大学);BIGAI(北京通用人工智能研究院);University of Toronto(多伦多大学);McGill University, Mila(麦吉尔大学,米拉);Nvidia Research(英伟达研究院);Amii(阿尔伯塔机器智能研究所);CIFAR AI Chair(CIFAR人工智能教席)
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Carnegie Mellon University(卡内基梅隆大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Meta AI
Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Kuaishou Technology(快手科技)
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Beihang University(北京航空航天大学);Zhongguancun Academy(中关村学院);Communication University of China(中国传媒大学);Peking University(北京大学);Hangzhou Innovation Institute of Beihang University(北京航空航天大学杭州创新研究院)