RollArt: 可分解的多任务智能体强化学习规模化训练
RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
AI总结：
提出RollArt系统，通过将强化学习流水线分解到异构硬件上，实现多任务智能体RL的高效训练，相比现有系统减少1.31-2.05倍训练时间。
摘要：
智能体强化学习通过与环境的多轮交互训练大语言模型，产生混合计算密集型预填充、带宽密集型解码、CPU密集型环境执行和突发性奖励评估的工作负载。现有系统要么将所有阶段共置于单一GPU集群，要么仅以粗粒度解耦，忽视了硬件异构性并导致阶段间大量同步开销。我们提出ROLLART，一个在可分解基础设施上的多任务智能体RL系统。ROLLART将每个流水线阶段映射到最合适的硬件：将预填充密集型任务路由到计算优化GPU，解码密集型任务路由到带宽优化GPU，环境任务路由到CPU集群。它在轨迹级别解耦生成，使得生成、环境交互和奖励评分可以独立进行，从而慢速或失败的环境不会阻塞其他任务。ROLLART将无状态奖励计算卸载到无服务器基础设施，并通过有界陈旧性的异步权重同步将生成与训练重叠。结果表明，ROLLART有效提高了训练吞吐量，与各种RL系统相比实现了1.31-2.05倍的训练时间减少。我们还在阿里巴巴集群上使用超过3000个GPU训练了用于Qoder产品的数千亿参数MoE模型，验证了其稳定性和可扩展性。
英文摘要：
Agentic Reinforcement Learning (RL) trains LLMs through multi-turn interactions with environments, producing workloads that mix compute-bound prefill, bandwidth-bound decoding, CPU-heavy environment execution, and bursty reward evaluation. Existing systems either colocate all stages on a single GPU cluster or decouple them only at a coarse granularity, overlooking hardware heterogeneity and incurring substantial synchronization overhead across stages. We present ROLLART, a system for multi-task agentic RL on disaggregated infrastructure. ROLLART maps each pipeline stage to best-fit hardware, routing prefill-heavy tasks to compute-optimized GPUs, decode-heavy tasks to bandwidth-optimized GPUs, and environments to CPU clusters. It decouples rollout at the trajectory level, allowing generation, environment interaction, and reward scoring to proceed independently, so that slow or failed environments never block the others. ROLLART offloads stateless reward computation to serverless infrastructure and overlaps rollout with training via staleness-bounded asynchronous weight synchronization. Our results demonstrate that ROLLART effectively improves training throughput and achieves 1.31--2.05 \(\times\) training time reduction compared to various RL systems. We also evaluated ROLLART by training a hundreds-of-billions-parameter MoE model for Qoder product on an Alibaba cluster with above 3,000 GPUs, demonstrating its stability and scalability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
