面向前沿大语言模型(LLMs)的生产级高效智能体强化学习(Agentic RL)框架及经验证的方案
LoongSage: A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes
AI总结:
针对前沿LLM智能体任务中现有RL框架笨重的问题,提出生产级高效智能体RL框架LoongSage,其基于Megatron、SGLang等构建,经评估可提升任务奖励并维持硬件利用率。
摘要:
强化学习(RL)已成为大语言模型(LLMs)后训练流程中的关键阶段。随着LLM能力前沿从静态推理转向智能体任务——其特征为外部工具调用、多轮交互及长 horizon 决策——现有主流RL框架愈发笨重,提升了采用、使用及调试的开销。为应对这些挑战,我们提出LoongSage,这是一款智能体原生的全异步RL框架,核心代码约1.4万行。该框架基于Megatron进行分布式训练,依托SGLang实现高吞吐量推理,通过单一控制器进程集中编排,并将所有驻留GPU的组件封装为Ray actors。它引入AgentFlow这一统一执行范式,用于编排白盒与黑盒智能体;针对异步执行,集成了限制轨迹陈旧性的滑动窗口策略,并采用TransferMesh这一拓扑感知同步机制,在每一步后通过各接收方可用的最快传输路由权重。除智能体RL外,LoongSage还包含内存高效的Teacher Manager,该模块将教师模型的扩展与GPU内存约束解耦,原生支持多教师在线蒸馏。在涵盖复杂强化学习及多教师蒸馏的四次完整训练运行中开展的广泛评估表明,任务奖励稳步提升,同时训练-推理差距得到持续控制,且全异步流水线消除了同步空闲气泡,维持了硬件利用率。
英文摘要:
Reinforcement learning (RL) has emerged as a critical phase in the post-training pipeline of large language models (LLMs). As the frontier of LLM capabilities shifts from static reasoning to agentic tasks--characterized by external tool invocation, multi-turn interactions, and long-horizon decision-making--existing mainstream RL frameworks have grown increasingly unwieldy, raising the overhead of adoption, usage, and debugging. To address these challenges, we present LoongSage, an agent-native, fully asynchronous RL framework with a lightweight core of roughly 14K lines of code. Built upon Megatron for distributed training and SGLang for high-throughput inference, LoongSage centralizes orchestration within a single controller process and encapsulates all GPU-resident components as Ray actors. It introduces AgentFlow, a unified execution paradigm that orchestrates both white-box and black-box agents. For asynchronous execution, it integrates sliding-window strategies that bound trajectory staleness, and employs TransferMesh, a topology-aware synchronization mechanism that routes weights after each step over the fastest transport available to each receiver. Beyond agentic RL, LoongSage incorporates a memory-efficient Teacher Manager that decouples the scaling of teacher models from GPU memory constraints, natively supporting Multi-Teacher On-Policy Distillation. Extensive evaluations across four complete training runs, spanning complex reinforcement learning and multi-teacher distillation, demonstrate that task rewards improve steadily while the train-inference gap is kept under continuous control, and the fully asynchronous pipeline removes synchronous idle bubbles, sustaining hardware utilization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。