arXiv 2510.042062025-10-05cs.AI

AgentRL：通过多轮多任务框架规模化实现智能体强化学习

AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework

Tsinghua University（清华大学）

Hanchen Zhang, Xiao Liu, Bowen Lv, Xueqiao Sun, Bohao Jing, Iat Long Iong, Zhenyu Hou, Zehan Qi, Hanyu Lai, Yifan Xu, Rui Lu, Hongning Wang, Jie Tang, Yuxiao Dong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多轮多任务场景下LLM智能体强化学习训练缺乏可扩展设施与稳定算法的问题，研究提出AgentRL框架，通过异步流水线、统一接口及跨策略采样等技术，在五项任务上超越多款主流LLM智能体，且已开源并应用于AutoGLM构建。

摘要：

大语言模型（LLM）的最新进展引发了学界对构建可通过在线交互学习的通用智能体的日益浓厚兴趣。然而，由于缺乏可扩展的基础设施和稳定的训练算法，将强化学习（RL）应用于多轮、多任务场景下的LLM智能体训练仍颇具挑战。本研究提出了AgentRL框架，用于可扩展的多轮多任务智能体强化学习训练。在基础设施层面，AgentRL采用全异步生成-训练流水线，以实现高效的多轮强化学习。为支持多任务强化学习中的异构环境开发，我们设计了基于函数调用的统一API接口、容器化环境开发方案以及中心化控制器。在算法层面，我们提出跨策略采样方法以促进多轮场景下的模型探索，并提出任务优势归一化方法以稳定多任务训练。实验表明，基于开源LLM在五项智能体任务上训练的AgentRL，性能显著优于GPT-5、Claude-Sonnet-4、DeepSeek-R1及其他开源LLM智能体。使用AgentRL进行的多任务训练效果与所有任务专属模型中的最优结果相当。AgentRL已在https://github.com/THUDM/AgentRL 开源，其算法与框架已被用于构建AutoGLM（https://autoglm.zhipuai.cn）。

英文摘要：

Recent advances in large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions. However, applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms. In this work, we present the AgentRL framework for scalable multi-turn, multi-task agentic RL training. On the infrastructure side, AgentRL features a fully-asynchronous generation-training pipeline for efficient multi-turn RL. To support heterogeneous environment development in multi-task RL, we design a unified function-call based API interface, containerized environment development, and a centralized controller. On the algorithm side, we propose cross-policy sampling to encourage model exploration in multi-turn settings and task advantage normalization to stabilize multi-task training. Experiments show that AgentRL, trained on open LLMs across five agentic tasks, significantly outperforms GPT-5, Clause-Sonnet-4, DeepSeek-R1, and other open-source LLM agents. Multi-task training with AgentRL matches the best results among all task-specific models. AgentRL is open-sourced at https://github.com/THUDM/AgentRL. The algorithm and framework are adopted in building \textsc{\href{https://autoglm.zhipuai.cn}{AutoGLM}}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。