Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, methods such as GRPO and DAPO suffer from substantial computational cost, since they rely on sampling many rollouts for each prompt. Moreover, in RLVR the relative advantage is often sparse: many samples become nearly all-correct or all-incorrect, yielding low within-group reward variance and thus weak learning signals. In this paper, we introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method that prunes rollouts during generation while explicitly steering the surviving ones more correctness-balanced to enhance learning signals. Specifically, arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions. The learned quality head can further weigh candidates to improve inference accuracy during test-time scaling. To improve efficiency, we present a system design that prunes rollouts inside the inference engine and re-batches the remaining ones for log-probability computation and policy updates. Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models (1B-8B), arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup, and yielding up to +8.33 additional gains in average accuracy in test-time scaling. The code is available at https://github.com/Hsu1023/ARRoL.
Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);Mila - Quebec AI Institute(Mila - 魁北克人工智能研究所);University of Waterloo(滑铁卢大学);The Chinese University of Hong Kong(香港中文大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Qwen Large Model Application Team, Alibaba Institute of Automation, Chinese Academy of Sciences(文勤大模型应用团队,阿里巴巴自动化研究所,中国科学院);Qwen Large Model Application Team, Alibaba(文勤大模型应用团队,阿里巴巴)
TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
University of Chinese Academy of Sciences(中国科学院大学);Meituan(美团);MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别国家重点实验室与智能信息处理重点实验室)
EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
The Chinese University of Hong Kong(香港中文大学);Microsoft Research Asia(微软亚洲研究院);Hong Kong Baptist University(香港浸会大学);LMU Munich(慕尼黑大学);MoE Key Laboratory of High Confidence Software Technologies(教育部高可信软件技术重点实验室)
Auditing Data Membership in Reinforcement Learning With Verifiable Rewards
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shanghai Jiao Tong University(上海交通大学);Shandong University(山东大学);Xidian University(西安电子科技大学);Wuhan University(武汉大学)
Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck
School of Engineering, Institute of Science Tokyo(科学东京大学工程学院);College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);Department of Electrical and Computer Engineering, National University of Singapore(新加坡国立大学电气与计算机工程学院)
JudgeRLVR: Judge First, Generate Second for Efficient Reasoning
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);CFCS, School of Computer Science, Peking University(计算机学院,北京大学);LLM-Core Xiaomi(小智LLM核心)
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
LMU Munich(慕尼黑大学);Harvard University(哈佛大学);University of Cambridge(剑桥大学);Mina AI;Konrad Zuse School of Excellence in Reliable AI (relAI)(康拉德·楚泽可靠人工智能卓越学校(relAI))
Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance
School of Artificial Intelligence, Beihang University, Beijing, China(北京航空航天大学人工智能学院);Hangzhou International Innovation Institute, Beihang University, Hangzhou, China(北京航空航天大学杭州国际创新研究院);University of Leicester, Leicester, United Kingdom(莱斯特大学);Nanyang Technological University, Singapore(南洋理工大学);The Hong Kong Polytechnic University, Hong Kong SAR, China(香港理工大学)