Over these years, multi-agent reinforcement learning has achieved remarkable performance in multi-agent planning and scheduling tasks. It typically follows the self-play setting, where agents are trained by playing with a fixed group of agents. However, in the face of zero-shot coordination, where an agent must coordinate with unseen partners, self-play agents may fail. Several methods have been proposed to handle this problem, but they either take a lot of time or lack generalizability. In this paper, we firstly reveal an important phenomenon: the zero-shot coordination performance is strongly linearly correlated with the similarity between an agent's training partner and testing partner. Inspired by it, we put forward a Similarity-Based Robust Training (SBRT) scheme that improves agents' zero-shot coordination performance by disturbing their partners' actions during training according to a pre-defined policy similarity value. To validate its effectiveness, we apply our scheme to three multi-agent reinforcement learning frameworks and achieve better performance compared with previous methods.
K-level Reasoning for Zero-Shot Coordination in Hanabi
Facebook AI Research（脸书人工智能研究院）；University of Oxford（牛津大学）
在合作的多代理设置的标准问题设置是自我发挥（SP），其目标是训练一个团队的代理一起工作。然而，最佳SP策略通常包含任意约定（“握手”），并且与其他独立训练的代理或人类不兼容。后一种需求最近由Hu等人（2020）正式定义为zero-shot协调（ZSC）设置，并部分通过他们的模拟游戏（OP）算法解决，该算法在纸牌游戏Hanabi中显示了改进的ZSC和人类AI性能。OP假设可以访问环境的对称性，并防止代理在训练期间以相互不兼容的方式破坏这些对称性。然而，正如作者所指出的，发现给定环境的对称性是一个计算上的难题。相反，我们表明，通过简单的k级推理（KLR）Costa Gomes et al. 2006，同步训练所有级别，我们可以在Hanabi中获得有竞争力的ZSC和ad-hoc团队合作性能，包括与类人代理机器人配对时。我们还介绍了一种新的方法，同步k级推理与最佳响应（SyKLRBR），它进一步提高了我们的同步KLR的性能，通过共同训练的最佳响应。
PC3D: Zero-Shot Cooperation Across Variable Rosters via Personalized Context Distillation
Doctoral School of Exact and Natural Sciences, Jagiellonian University（杰哥利昂大学精确与自然科学博士学院）；Faculty of Mathematics and Computer Science, Jagiellonian University（杰哥利昂大学数学与计算机科学学院）
