arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08272cs.LGcs.AI
收藏

多智能体系统何时表现更优?分析智能体系统的学习效率

When Do Multi-Agent Systems Outperform? Analysing the Learning Efficiency of Agentic Systems

Department of Computer Science, University of Hong Kong(计算机科学系,香港大学)

Junwei Su, Chuan Wu

AI总结:

本文研究多智能体强化学习在大语言模型中的学习效率,通过理论分析揭示任务分解与对齐对样本复杂性的影响,明确MARL在特定任务下的优势条件。

摘要:

强化学习(RL)已作为一种关键方法用于训练或微调大语言模型(LLMs),通过交互反馈实现适应性、任务特定的优化。多智能体强化学习(MARL)尤其提供了一个有前途的途径,通过将复杂任务分解为由不同相互作用的智能体学习的专用子任务,可能提高LLM系统的能力和效率。然而,关于何时以及为何MARL优于单智能体RL(SARL)的理论见解仍然有限,这在选择适当的RL框架时造成了不确定性。在本文中,我们通过严格分析MARL和SARL在LLM上下文中的比较样本效率来解决这一关键缺口。利用Probably Approximately Correct(PAC)框架,我们正式定义了LLM中的SARL和MARL设置,推导出显式的样本复杂性界限,并系统地刻画任务分解和对齐如何影响学习效率。我们的结果表明,当任务自然分解为独立的子任务时,MARL会提高样本复杂性,而依赖性子任务会削弱MARL的比较优势。此外,我们引入并分析了任务对齐的概念,量化了在存在潜在不一致的情况下强制独立任务分解的权衡。这些理论见解澄清了经验不一致,并为在复杂的LLM场景中有效部署MARL策略提供了实际标准。

英文摘要:

Reinforcement Learning (RL) has emerged as a crucial method for training or fine-tuning large language models (LLMs), enabling adaptive, task-specific optimizations through interactive feedback. Multi-Agent Reinforcement Learning (MARL), in particular, offers a promising avenue by decomposing complex tasks into specialized subtasks learned by distinct interacting agents, potentially enhancing the ability and efficiency of LLM systems. However, theoretical insights regarding when and why MARL outperforms Single-Agent RL (SARL) remain limited, creating uncertainty in selecting the appropriate RL framework. In this paper, we address this critical gap by rigorously analyzing the comparative sample efficiency of MARL and SARL within the context of LLM. Leveraging the Probably Approximately Correct (PAC) framework, we formally define SARL and MARL setups for LLMs, derive explicit sample complexity bounds, and systematically characterize how task decomposition and alignment influence learning efficiency. Our results demonstrate that MARL improves sample complexity when tasks naturally decompose into independent subtasks, whereas dependent subtasks diminish MARL's comparative advantage. Additionally, we introduce and analyze the concept of task alignment, quantifying the trade-offs when enforcing independent task decomposition despite potential misalignments. These theoretical insights clarify empirical inconsistencies and provide practical criteria for deploying MARL strategies effectively in complex LLM scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑