强化学习(RL)已作为一种关键方法用于训练或微调大语言模型(LLMs),通过交互反馈实现适应性、任务特定的优化。多智能体强化学习(MARL)尤其提供了一个有前途的途径,通过将复杂任务分解为由不同相互作用的智能体学习的专用子任务,可能提高LLM系统的能力和效率。然而,关于何时以及为何MARL优于单智能体RL(SARL)的理论见解仍然有限,这在选择适当的RL框架时造成了不确定性。在本文中,我们通过严格分析MARL和SARL在LLM上下文中的比较样本效率来解决这一关键缺口。利用Probably Approximately Correct(PAC)框架,我们正式定义了LLM中的SARL和MARL设置,推导出显式的样本复杂性界限,并系统地刻画任务分解和对齐如何影响学习效率。我们的结果表明,当任务自然分解为独立的子任务时,MARL会提高样本复杂性,而依赖性子任务会削弱MARL的比较优势。此外,我们引入并分析了任务对齐的概念,量化了在存在潜在不一致的情况下强制独立任务分解的权衡。这些理论见解澄清了经验不一致,并为在复杂的LLM场景中有效部署MARL策略提供了实际标准。
英文摘要:
Reinforcement Learning (RL) has emerged as a crucial method for training or fine-tuning large language models (LLMs), enabling adaptive, task-specific optimizations through interactive feedback. Multi-Agent Reinforcement Learning (MARL), in particular, offers a promising avenue by decomposing complex tasks into specialized subtasks learned by distinct interacting agents, potentially enhancing the ability and efficiency of LLM systems. However, theoretical insights regarding when and why MARL outperforms Single-Agent RL (SARL) remain limited, creating uncertainty in selecting the appropriate RL framework. In this paper, we address this critical gap by rigorously analyzing the comparative sample efficiency of MARL and SARL within the context of LLM. Leveraging the Probably Approximately Correct (PAC) framework, we formally define SARL and MARL setups for LLMs, derive explicit sample complexity bounds, and systematically characterize how task decomposition and alignment influence learning efficiency. Our results demonstrate that MARL improves sample complexity when tasks naturally decompose into independent subtasks, whereas dependent subtasks diminish MARL's comparative advantage. Additionally, we introduce and analyze the concept of task alignment, quantifying the trade-offs when enforcing independent task decomposition despite potential misalignments. These theoretical insights clarify empirical inconsistencies and provide practical criteria for deploying MARL strategies effectively in complex LLM scenarios.
RewardFlow: Can Action-Level Reward Be Estimated from State-Graph Topology in Agentic RL?
TMLR Group(TMLR小组);Hong Kong Baptist University(香港 Baptist大学);TCL Corporate Research (HK) Co Ltd(TCL企业研究(香港)有限公司);Cooperative Medianet Innovation Center Shanghai Jiao Tong University(合作中位网创新中心上海交通大学);Department of Mathematics Hong Kong Baptist University(香港 Baptist大学数学系)
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DeepWisdom;Singapore University of Technology and Design(新加坡科技设计大学);Nanyang Technological University(南洋理工大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Université de Montréal & Mila(蒙特利尔大学及Mila)
AMEM4Rec: Leveraging Cross-User Similarity for Memory Evolution in Agentic LLM Recommenders
Center for AI Research, VinUniversity(越南 Vin 大学人工智能研究中心);College of Engineering and Computer Science, VinUniversity(越南 Vin 大学工程与计算机科学学院);Computer Science, VinUniversity(越南 Vin 大学计算机科学)