Despite algorithm-level innovations for multi-agent reinforcement learning (MARL), the underlying infrastructure for large-scale MARL remains underexplored. Existing training frameworks primarily optimize for single-agent scenarios and fail to capture the graph-structured nature of MARL. This fundamental mismatch leads to three MARL-specific challenges: topology-induced rollout skew, critical path synchronization barriers, and dynamic training demand. To bridge this gap, we propose FlexMARL, a graph-driven rollout-training co-designed framework for large-scale LLM-based MARL. FlexMARL introduces the multi-agent execution graph (MEG) and materializes the intermediate representation as the MEG Store, which provides decentralized multi-table storage and serves as the data hub for system components. Built around the MEG Store, the Rollout Engine develops dependency parallel sampling with hierarchical load balancing, which reduces inference latency and balances intra-agent/inter-agent loads. The Joint Orchestrator exploits an agent-level asynchronous pipeline to mitigate the critical path synchronization barriers while maintaining consistency. The Training Engine achieves on-demand resource allocation by collectively managing training processes for individual agents. The training states of different agents are swapped via location-agnostic communication APIs. Empirical results on a production cluster demonstrate that FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks.
SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
Cornell University（康奈尔大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Tsinghua University（清华大学）；University of Washington（华盛顿大学）；ByteDance（字节跳动）
Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models
Seoul National University（首尔大学）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能交叉项目）；Artificial Intelligence Institute, Seoul National University（首尔大学人工智能研究院）；Department of Intelligence and Information, Seoul National University（首尔大学智能与信息系）
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Beihang University（北京航空航天大学）；Zhongguancun Academy（中关村学院）；Communication University of China（中国传媒大学）；Peking University（北京大学）；Hangzhou Innovation Institute of Beihang University（北京航空航天大学杭州创新研究院）
ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts
National Key Laboratory for Novel Software Technology Nanjing University China（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence Nanjing University China（南京大学人工智能学院）；The University of Hong Kong（香港大学）
SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanyang Technological University（南洋理工大学）
Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Huawei（华为）
RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Southern University of Science and Technology（南方科技大学）
EchoRL: Reinforcement Learning via Rollout Echoing
Munich Center for Machine Learning（慕尼黑机器学习中心）；Huawei Heisenberg Research Center（华为海森堡研究所以）；University of Arizona（亚利桑那大学）；College of Computing（计算学院）；Data Science, Nanyang Technological University, Singapore（数据科学，南洋理工大学，新加坡）；MemAgents Lab（MemAgents实验室）
Learning from Hindsight for VLA Reinforcement Learning
Massachusetts Institute of Technology（麻省理工学院）；MIT-IBM Computing Research Lab（麻省理工学院-IBM计算研究实验室）；Stanford University（斯坦福大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Carnegie Mellon University（卡内基梅隆大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Meta AI
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算与智能学院）；Beijing Zhongguancun Academy（北京中关村学院）；XinzhuAI（芯智科技）
ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
The University of Hong Kong（香港大学）；Fudan University（复旦大学）；Gradient；University of Edinburgh（爱丁堡大学）；Soochow University（苏州大学）；Technical University of Darmstadt（达姆施塔特技术大学）；University of the Chinese Academy of Sciences（中国科学院大学）
RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
Shanghai Jiao Tong University（上海交通大学）；Central South University（中南大学）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
School of Computer Science \& Beijing Key Laboratory of Software；Hardware Cooperative Artificial Intelligence Systems, Peking University, Beijing, China；School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China；Institute of Computational Social Science, Peking University (Qingdao), Qingdao, China
Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
School of Information, Renmin University of China（中国人民大学信息学院）；Key Laboratory of Data Engineering and Knowledge Engineering（数据工程与知识工程重点实验室）；Ant Group（蚂蚁集团）
APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
Advanced Micro Devices, Inc. (AMD)（先进微器件公司）；Carnegie Mellon University (CMU)（卡内基梅隆大学）；LMSYS Org（LMSYS组织）；University of California, Los Angeles (UCLA)（加州大学洛杉矶分校）
VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
National Key Laboratory for Novel Software Technology, Nanjing University, China \& School of Artificial Intelligence, Nanjing University, China；Computational Intelligence Center (CIC), School of Computer；Artificial Intelligence, Shandong Jianzhu University, Jinan, 250101, China
