AgentFly：面向语言模型智能体的可扩展且可伸缩的强化学习框架
AgentFly: Scaling Agentic Reinforcement Learning with Unified Resource System
AI总结：
AgentFly是一个可扩展的Agent-RL框架，通过令牌级掩码、装饰器接口和异步执行机制，为语言模型智能体提供多种强化学习算法支持，并在多个任务上验证了其有效性。
摘要：
语言模型（LM）智能体因其能够通过与环境、工具和API交互自主完成任务而受到广泛关注。LM智能体主要通过提示工程或监督微调来构建。与此同时，强化学习（RL）已被探索用于增强LM的能力，例如推理能力和事实性。然而，LM智能体与强化学习的结合（Agent-RL）仍然未被充分探索，且缺乏系统性研究。为此，我们构建了AgentFly，一个可扩展且可伸缩的Agent-RL框架，旨在为LM智能体赋能多种强化学习算法。我们的框架通过采用令牌级掩码来适配传统RL方法，从而支持多轮交互。该框架具有基于装饰器的接口，用于定义工具和奖励函数，实现了无缝扩展和易用性。为了支持高吞吐量训练，我们实现了工具调用和奖励计算的异步执行，并设计了一个集中式资源管理系统，用于可伸缩的环境协调。我们还提供了一套预构建的工具和环境，通过在多个任务上成功训练智能体，展示了该框架的有效性。
英文摘要：
Methods to build LLM agents have evolved from prompt engineering and supervised finetuning to agentic reinforcement learning (agentic RL). However, agentic RL remains bottlenecked by its surrounding systems: agents must interact with heterogeneous environments, such as sandboxes, model services, and external APIs. Their allocation, reuse, and lifecycle dominate rollout cost and cap the scale at which training becomes practical. In this work, we present AgentFly, an agentic RL framework built with a unified resource layer that treats each of these environments as a distinct, typed resource scheduled through one engine, with per-tool acquisition for multi-turn reuse, asynchronous backpressure, and rollout versus global-scoped lifecycles. AgentFly adopts a four-layer design: (I) agent layer that abstracts the agent, tool, and reward concepts, decomposing agentic RL into defining agents, tools, and reward functions; (II) rollout layer that composes these into agent loops and computes rewards; (III) context layer that organizes rollouts, injects contextual information, and arranges resources; and (IV) a low-level resource layer that performs resource management. We provide a suite of prebuilt tools and environments, demonstrate successful agent training across multiple tasks and models, and report the first controlled cross-framework throughput comparison against agentic RL frameworks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
