arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.08184cs.AIcs.LG
收藏

通过场景无关表示实现多智能体迁移强化学习

Enabling Multi-Agent Transfer Reinforcement Learning via Scenario Independent Representation

Missouri State University(密苏里州立大学)

Ayesha Siddika Nipu, Siming Liu, Anthony Harris

AI总结:

提出一种通过统一状态空间实现多智能体迁移强化学习的框架,在SMAC环境中验证了其能显著提升学习性能,并结合课程迁移学习促进复杂异构场景下的知识迁移。

摘要:

多智能体强化学习(MARL)算法被广泛用于解决动态多智能体系统(MAS)中需要智能体协作与竞争的复杂任务。然而,从零开始学习此类任务十分艰巨且并非总是可行,特别是对于具有大量交互智能体的MAS,因为其样本复杂度极高。因此,重用从过去经验或其他智能体获得的知识,能够有效加速学习过程并提升MARL算法的规模。在本研究中,我们提出了一种新颖的框架,通过将各种状态空间统一为固定大小的输入,使统一的深度学习策略能在MAS的不同场景中适用,从而实现MARL的迁移学习。我们在StarCraft Multi-Agent Challenge(SMAC)环境的一系列场景中评估了该方法,结果表明,与从零开始学习的智能体相比,利用从其他场景学到的机动技能显著提升了多智能体学习性能。此外,我们采用了课程迁移学习(CTL),使深度学习策略能够按难度级别组织的预设同质学习场景中,逐步获取知识与技能。该过程促进了智能体间及智能体内部的知识迁移,从而在更复杂的异构场景中实现了高多智能体学习性能。

英文摘要:

Multi-Agent Reinforcement Learning (MARL) algorithms are widely adopted in tackling complex tasks that require collaboration and competition among agents in dynamic Multi-Agent Systems (MAS). However, learning such tasks from scratch is arduous and may not always be feasible, particularly for MASs with a large number of interactive agents due to the extensive sample complexity. Therefore, reusing knowledge gained from past experiences or other agents could efficiently accelerate the learning process and upscale MARL algorithms. In this study, we introduce a novel framework that enables transfer learning for MARL through unifying various state spaces into fixed-size inputs that allow one unified deep-learning policy viable in different scenarios within a MAS. We evaluated our approach in a range of scenarios within the StarCraft Multi-Agent Challenge (SMAC) environment, and the findings show significant enhancements in multi-agent learning performance using maneuvering skills learned from other scenarios compared to agents learning from scratch. Furthermore, we adopted Curriculum Transfer Learning (CTL), enabling our deep learning policy to progressively acquire knowledge and skills across pre-designed homogeneous learning scenarios organized by difficulty levels. This process promotes inter- and intra-agent knowledge transfer, leading to high multi-agent learning performance in more complicated heterogeneous scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑