第二部分：ROLL Flash——通过异步加速 RLVR 与智能体训练
Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
AI总结：
针对同步强化学习后训练资源利用率低和可扩展性受限的问题，提出基于细粒度并行和 rollout-train 解耦的异步系统 ROLL Flash，在同等 GPU 预算下于 RLVR 与智能体任务上实现显著加速且性能与同步训练相当。
摘要：
同步强化学习（RL）后训练已成为增强大型语言模型（LLM）多样化能力的关键步骤。然而，许多旨在加速 RL 后训练的系统仍面临资源利用率低和可扩展性有限的问题。我们提出了 ROLL Flash，该系统通过原生支持异步 RL 后训练来扩展 ROLL。ROLL Flash 基于两个核心设计原则：细粒度并行性和 rollout-train 解耦。在这些原则的指导下，ROLL Flash 提供了灵活的编程接口，支持完全异步的训练架构，并支持高效的 rollout 机制，包括队列调度和环境级异步执行。通过全面的理论分析和大量实验，我们证明 ROLL Flash 相比同步 RL 后训练显著提升了资源利用率和可扩展性。在使用与同步基线相同的 GPU 预算下，ROLL Flash 在 RLVR 任务上实现了高达 2.24 倍的加速，在智能体任务上实现了 2.72 倍的加速。此外，我们实现了几种流行的 off-policy 算法，并验证了异步训练能够达到与同步训练相当的性能。
英文摘要：
Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems designed to accelerate RL post-training still suffer from low resource utilization and limited scalability. We present ROLL Flash, a system that extends ROLL with native support for asynchronous RL post-training. ROLL Flash is built upon two core design principles: fine-grained parallelism and rollout-train decoupling. Guided by these principles, ROLL Flash provides flexible programming interfaces that enable a fully asynchronous training architecture and support efficient rollout mechanisms, including queue scheduling and environment-level asynchronous execution. Through comprehensive theoretical analysis and extensive experiments, we demonstrate that ROLL Flash significantly improves resource utilization and scalability over synchronous RL post-training. ROLL Flash achieves up to 2.24x speedup on RLVR tasks and 2.72x on agentic tasks, using the same GPU budget as synchronous baselines. Furthermore, we implement several popular off-policy algorithms and verify that asynchronous training can achieve performance on par with synchronous training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
