arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08847cs.LGcs.AI
收藏

Dr. MAS:多智能体大语言模型系统的稳定强化学习

Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

Nanyang Technological University, Singapore(南洋理工大学,新加坡)

Lang Feng, Longtao Zheng, Shuo He, Fuxiang Zhang, Bo An

AI总结:

Dr. MAS通过智能体级归一化方法稳定多智能体大语言模型的强化学习训练,提升性能并减少梯度不稳定问题。

摘要:

多智能体大语言模型系统通过角色专业化实现了高级推理和工具使用,但此类系统在训练后的可靠强化学习(RL)仍极具挑战性。在本工作中,我们从理论上指出了将基于群体的RL扩展到多智能体大语言模型系统时训练不稳定的关键原因。我们证明,在GRPO风格的优化下,一个全局归一化基线可能偏离不同智能体的奖励分布,最终导致梯度范数不稳定。基于这一发现,我们提出了Dr. MAS,一种用于多智能体大语言模型系统的简单且稳定的RL训练方法。Dr. MAS采用一种智能体级别的解决方案:使用每个智能体自身的奖励统计数据对优势进行归一化,这校准了梯度尺度并显著稳定了训练,理论和实践上均有效。除了算法外,Dr. MAS还为多智能体大语言模型系统提供了一个端到端的RL训练框架,支持可扩展的编排、灵活的每智能体大语言模型服务和优化配置,以及共享资源调度的大语言模型动作后端。我们评估了Dr. MAS在多智能体数学推理和多轮搜索基准上使用Qwen2.5和Qwen3系列模型的表现。Dr. MAS在数学和搜索基准上分别实现了比原生GRPO显著的提升(例如,在数学上平均@16提升了5.6%,通过@16提升了4.6%;在搜索上平均@16提升了15.2%,通过@16提升了13.1%),同时大幅减少了梯度尖峰。此外,它在异构智能体-模型分配下仍保持高度有效并提高了效率。

英文摘要:

Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6\% avg@16 and +4.6\% pass@16 on math, and +15.2\% avg@16 and +13.1\% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑