有状态主动促进器：协作多智能体强化学习中的协调与环境异质性
Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
AI总结：
该研究针对协作多智能体强化学习中环境异质性与协调需求差异的问题，提出了可定量控制异质性和协调水平的HECOGrid环境套件，以及基于共享知识源和策略池动态选择的SAF方法，实验证明其性能优于IPPO、MAPPO基线。
摘要：
在协作多智能体强化学习中，一组智能体协同工作以实现共同目标。不同的环境或任务可能需要智能体之间具备不同程度的协调能力，才能以最优方式达成目标。协调的特性取决于环境的属性——包括空间布局、障碍物分布、动态特性等。我们将环境内部的这种属性差异称为异质性。现有研究尚未充分解决不同环境可能具有不同异质性水平这一问题。我们对环境的协调水平和异质性水平的概念进行了形式化定义，并提出了HECOGrid（异质性协调网格）——一套多智能体强化学习环境套件，它通过对环境的协调水平和异质性水平提供定量控制，便于对不同多智能体强化学习方法在不同协调与环境异质性水平下的表现进行实证评估。此外，我们提出了一种名为Stateful Active Facilitator（SAF，有状态主动促进器）的集中式训练分布式执行学习方法，该方法通过训练期间使用的可微共享知识源，以及从共享策略池中进行动态选择，使智能体能够在高协调、高异质性环境中高效工作。我们在HECOGrid上对SAF进行了评估，并将其性能与基线方法IPPO和MAPPO进行了对比。结果表明，在不同任务以及不同异质性和协调水平下，SAF的表现始终优于基线方法。我们已开源HECOGrid的代码以及所有实验的相关代码。
英文摘要：
In cooperative multi-agent reinforcement learning, a team of agents works together to achieve a common goal. Different environments or tasks may require varying degrees of coordination among agents in order to achieve the goal in an optimal way. The nature of coordination will depend on the properties of the environment -- its spatial layout, distribution of obstacles, dynamics, etc. We term this variation of properties within an environment as heterogeneity. Existing literature has not sufficiently addressed the fact that different environments may have different levels of heterogeneity. We formalize the notions of coordination level and heterogeneity level of an environment and present HECOGrid, a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by providing a quantitative control over coordination and heterogeneity levels of the environment. Further, we propose a Centralized Training Decentralized Execution learning approach called Stateful Active Facilitator (SAF) that enables agents to work efficiently in high-coordination and high-heterogeneity environments through a differentiable and shared knowledge source used during training and dynamic selection from a shared pool of policies. We evaluate SAF and compare its performance against baselines IPPO and MAPPO on HECOGrid. Our results show that SAF consistently outperforms the baselines across different tasks and different heterogeneity and coordination levels. We release the code for HECOGrid as well as all our experiments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
