arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.13019cs.DCcs.AIcs.LG
收藏

ARL-Tangram:释放代理强化学习中的资源效率

ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning

School of Computer Science, Peking University(北京大学计算机科学系);LLM-Core Xiaomi(小智LLM核心)

Bangjun Xiao, Yihao Zhao, Xiangwei Deng, Shihua Yu, Yuxing Xiang, Huaqiu Liu, Qiying Wang, Liang Zhao, Hailin Zhang, Xuanzhe Liu, Xin Jin, Fuli Luo

AI总结:

ARL-Tangram通过动作级 orchestration 实现细粒度外部资源共享与弹性调度,提升代理强化学习的资源利用效率,实验表明其在平均动作完成时间、训练步长速度和外部资源节约方面均有显著提升。

摘要:

代理强化学习(RL)已作为云集群中的变革性工作负载出现,使大语言模型(LLMs)通过与现实世界交互解决复杂问题。然而,与传统RL不同,代理RL需要大量外部云资源,如用于代码执行的CPU和用于奖励模型的GPU,这些资源存在于主要训练集群之外。现有代理RL框架通常依赖静态过分配,即资源常与长生命周期轨迹绑定或被任务隔离,导致严重的资源不效率。我们提出动作级 orchestration,并将其纳入ARL-Tangram,一个统一的资源管理系统,实现细粒度的外部资源共享和弹性。ARL-Tangram利用统一的动作级公式和弹性调度算法,以最小化动作完成时间(ACT)同时满足异构资源约束。此外,异构资源管理器被定制以高效支持具有异构特性和拓扑的资源上的动作级执行。在真实世界代理RL任务上的评估表明,ARL-Tangram将平均ACT提高高达4.3倍,将RL训练的步长持续时间加速高达1.5倍,并节省外部资源高达71.2%。该系统已部署以支持MiMo系列模型的训练。

英文摘要:

Agentic reinforcement learning (RL) has emerged as a transformative workload in cloud clusters, enabling large language models (LLMs) to solve complex problems through interactions with real world. However, unlike traditional RL, agentic RL demands substantial external cloud resources, e.g., CPUs for code execution and GPUs for reward models, that exist outside the primary training cluster. Existing agentic RL framework typically rely on static over-provisioning, i.e., resources are often tied to long-lived trajectories or isolated by tasks, which leads to severe resource inefficiency. We propose the action-level orchestration, and incorporate it into ARL-Tangram, a unified resource management system that enables fine-grained external resource sharing and elasticity. ARL-Tangram utilizes a unified action-level formulation and an elastic scheduling algorithm to minimize action completion time (ACT) while satisfying heterogeneous resource constraints. Further, heterogeneous resource managers are tailored to efficiently support the action-level execution on resources with heterogeneous characteristics and topologies. Evaluation on real-world agentic RL tasks demonstrates that ARL-Tangram improves average ACT by up to 4.3$\times$, speeds up the step duration of RL training by up to 1.5$\times$, and saves the external resources by up to 71.2$\%$. This system has been deployed to support the training of the MiMo series models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑