面向零样本可扩展协作的异构多智能体强化学习
Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration
AI总结：
本文提出SHPPO框架，通过隐变量网络和异构层为参数共享PPO引入个体间与时间异构性，在SMAC和GRF中实现零样本可扩展协作并提升性能。
摘要：
多智能体强化学习（MARL）的出现正在显著改变自动驾驶车辆网络等多个领域。然而，现实世界中的多智能体系统通常包含多种角色，且这些系统的规模会动态波动。因此，为了实现零样本可扩展协作，不同角色的策略必须能够根据规模灵活更新，这对当前的MARL框架而言仍然是一个挑战。为了解决这一问题，我们提出了一种名为可扩展异构近端策略优化（SHPPO）的新型MARL框架，将异构性集成到基于参数共享PPO的MARL网络中。我们首先利用一个隐变量网络自适应地学习每个智能体的策略模式。其次，我们引入一个异构层插入到决策网络中，其参数由学习到的隐变量专门生成。我们的方法具有可扩展性，因为除异构层外所有参数均共享，并且同时获得了个体间异构性和时间异构性，使SHPPO能够有效适应不同规模。SHPPO在经典MARL环境如星际争霸多智能体挑战（SMAC）和Google Research Football（GRF）中表现出优越性能，展示了增强的零样本可扩展性，并通过可视化揭示了学习到的隐变量对团队表现的影响。
英文摘要：
The emergence of multi-agent reinforcement learning (MARL) is significantly transforming various fields like autonomous vehicle networks. However, real-world multi-agent systems typically contain multiple roles, and the scale of these systems dynamically fluctuates. Consequently, in order to achieve zero-shot scalable collaboration, it is essential that strategies for different roles can be updated flexibly according to the scales, which is still a challenge for current MARL frameworks. To address this, we propose a novel MARL framework named Scalable and Heterogeneous Proximal Policy Optimization (SHPPO), integrating heterogeneity into parameter-shared PPO-based MARL networks. We first leverage a latent network to learn strategy patterns for each agent adaptively. Second, we introduce a heterogeneous layer to be inserted into decision-making networks, whose parameters are specifically generated by the learned latent variables. Our approach is scalable as all the parameters are shared except for the heterogeneous layer, and gains both inter-individual and temporal heterogeneity, allowing SHPPO to adapt effectively to varying scales. SHPPO exhibits superior performance in classic MARL environments like Starcraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF), showcasing enhanced zero-shot scalability, and offering insights into the learned latent variables' impact on team performance by visualization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
