Libra：面向智能体强化学习后训练的高效资源管理
Libra: Efficient Resource Management for Agentic RL Post-Training
AI总结：
针对智能体强化学习中长尾、非平稳工作负载带来的资源管理挑战，提出Libra系统，通过周期性全局资源规划器和因果驱动多级反馈队列调度器，实现GPU分配优化和请求调度，最高提升3倍吞吐量和2.5倍收敛速度。
摘要：
强化学习（RL）已成为大型语言模型（LLM）的标准后训练范式，从偏好对齐扩展到复杂推理和多轮智能体行为。在智能体RL中，rollout阶段生成轨迹并调用工具，产生长尾和非平稳的工作负载，挑战了传统的资源管理假设。出现了三个基本挑战。首先，由于长尾分布，一小部分轨迹主导了rollout完成时间。其次，rollout和训练在计算模式、内存需求和对序列长度的敏感性上表现出强烈的不对称性。第三，随着RL策略的演变，轨迹长度分布随时间漂移，使得任何静态资源分配逐渐变得次优。我们提出Libra，引入了两个核心机制。第一个是周期性全局资源规划器，它联合优化rollout和训练集群间的GPU分配。它利用弹性混合池实现阶段间轻量级、非阻塞的工作节点重新分配。第二个是因果驱动的多级反馈队列（C-MLFQ）调度器，它基于从工具返回结果导出的因果信号（而非依赖脆弱的长度的预测）将请求路由到异构的rollout桶。在48个A800 GPU上的评估表明，与基线相比，Libra实现了高达3.0倍的吞吐量提升和高达2.5倍的奖励收敛加速。
英文摘要：
Reinforcement learning (RL) has emerged as a standard post-training paradigm for shaping large language models (LLMs) into capable agents. In agentic RL, the rollout stage generates trajectories while invoking tools, producing long-tailed and non-stationary workloads that expose two fundamental challenges. First, due to the long-tailed response distribution, a small fraction of trajectories dominates rollout makespan.Second, rollout and training differ in their compute patterns, memory demands, and sensitivity to sequence length. As the policy evolves, shifts in the workload distribution further change their relative resource demands, making it difficult to maintain balanced execution across the two stages. We present Libra, an adaptive runtime for agentic RL post-training with two complementary components: (1) intra-stage scheduling via a Causality-Guided Bucket Scheduler that routes requests across execution buckets with different parallelism configurations, reducing delays from rollout stragglers; and (2) cross-stage coordination that dynamically reallocates workers between rollout and training as the workload changes. It moves workers between the two stages through a non-blocking protocol without interrupting ongoing training. Evaluated on a 48x NVIDIA A800 GPU cluster and a 160x Ascend 910B3 NPU cluster across three agentic benchmarks, Libra achieves up to 4.2x higher throughput and up to 2.7x faster reward convergence
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
