arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.18291cs.AI
收藏

扩散以协调:高效在线多智能体扩散策略

Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

University of Science and Technology of China(中国科学技术大学)

Zhuoran Li, Hai Zhong, Xun Wang, Qingxin Xia, Lihua Zhang, Longbo Huang

AI总结:

提出首个在线离线策略多智能体强化学习框架OMAD,利用扩散策略和松弛策略目标最大化缩放联合熵,实现高效探索与协调,在MPE和MAMuJoCo上样本效率提升2.5至5倍。

摘要:

在线多智能体强化学习(MARL)是实现高效智能体协调的重要框架。关键在于增强策略表达能力以实现更优性能。基于扩散的生成模型在图像生成和离线设置中展现出卓越的表达能力和多模态表示,因此非常适合满足这一需求。然而,它们在在线MARL中的潜力尚未被充分探索。主要障碍是扩散模型的难以处理的似然性阻碍了基于熵的探索和协调。为应对这一挑战,我们首次提出使用扩散策略的在线离线策略MARL框架(OMAD)来协调协调。我们的关键创新是采用松弛策略目标,最大化缩放联合熵,从而在无需可处理似然的情况下促进有效探索。此外,在集中训练与分散执行(CTDE)范式中,我们使用联合分布价值函数来优化分散扩散策略。它利用可处理的熵增强目标来指导扩散策略的同时更新,从而确保稳定协调。在MPE和MAMuJoCo上的广泛评估表明,我们的方法在10个不同任务上达到了新的最先进水平,样本效率显著提升了2.5至5倍。

英文摘要:

Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first \underline{O}nline off-policy \underline{MA}RL framework using \underline{D}iffusion policies (\textbf{OMAD}) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across $10$ diverse tasks, demonstrating a remarkable $2.5\times$ to $5\times$ improvement in sample efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑