ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
The University of Hong Kong（香港大学）；Fudan University（复旦大学）；Gradient；University of Edinburgh（爱丁堡大学）；Soochow University（苏州大学）；Technical University of Darmstadt（达姆施塔特技术大学）；University of the Chinese Academy of Sciences（中国科学院大学）
Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of LLMs ranging from 4B to 32B parameters under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.
EchoRL: Reinforcement Learning via Rollout Echoing
Munich Center for Machine Learning（慕尼黑机器学习中心）；Huawei Heisenberg Research Center（华为海森堡研究所以）；University of Arizona（亚利桑那大学）；College of Computing（计算学院）；Data Science, Nanyang Technological University, Singapore（数据科学，南洋理工大学，新加坡）；MemAgents Lab（MemAgents实验室）
RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Southern University of Science and Technology（南方科技大学）
SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
Cornell University（康奈尔大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Tsinghua University（清华大学）；University of Washington（华盛顿大学）；ByteDance（字节跳动）
Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models
Seoul National University（首尔大学）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能交叉项目）；Artificial Intelligence Institute, Seoul National University（首尔大学人工智能研究院）；Department of Intelligence and Information, Seoul National University（首尔大学智能与信息系）
SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanyang Technological University（南洋理工大学）
Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Huawei（华为）
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Beihang University（北京航空航天大学）；Zhongguancun Academy（中关村学院）；Communication University of China（中国传媒大学）；Peking University（北京大学）；Hangzhou Innovation Institute of Beihang University（北京航空航天大学杭州创新研究院）
ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts
National Key Laboratory for Novel Software Technology Nanjing University China（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence Nanjing University China（南京大学人工智能学院）；The University of Hong Kong（香港大学）
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算与智能学院）；Beijing Zhongguancun Academy（北京中关村学院）；XinzhuAI（芯智科技）
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Carnegie Mellon University（卡内基梅隆大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Meta AI
RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
Shanghai Jiao Tong University（上海交通大学）；Central South University（中南大学）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
School of Computer Science \& Beijing Key Laboratory of Software；Hardware Cooperative Artificial Intelligence Systems, Peking University, Beijing, China；School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China；Institute of Computational Social Science, Peking University (Qingdao), Qingdao, China
Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
School of Information, Renmin University of China（中国人民大学信息学院）；Key Laboratory of Data Engineering and Knowledge Engineering（数据工程与知识工程重点实验室）；Ant Group（蚂蚁集团）
Learning from Hindsight for VLA Reinforcement Learning
Massachusetts Institute of Technology（麻省理工学院）；MIT-IBM Computing Research Lab（麻省理工学院-IBM计算研究实验室）；Stanford University（斯坦福大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
