In this paper, we propose a new mutual information framework for multi-agent reinforcement learning to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the simultaneous mutual information between multi-agent actions. By introducing a latent variable to induce nonzero mutual information between multi-agent actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function. The derived tractable objective can be interpreted as maximum entropy reinforcement learning combined with uncertainty reduction of other agents actions. Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic, which follows centralized learning with decentralized execution. We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms other MARL algorithms in multi-agent tasks requiring high-quality coordination.
A New Formalism, Method and Open Issues for Zero-Shot Coordination
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of California, Berkeley（加州大学伯克利分校）；Duke University（杜克大学）；Facebook AI Research（脸书人工智能研究院）
K-level Reasoning for Zero-Shot Coordination in Hanabi
Facebook AI Research（脸书人工智能研究院）；University of Oxford（牛津大学）
在合作的多代理设置的标准问题设置是自我发挥（SP），其目标是训练一个团队的代理一起工作。然而，最佳SP策略通常包含任意约定（“握手”），并且与其他独立训练的代理或人类不兼容。后一种需求最近由Hu等人（2020）正式定义为zero-shot协调（ZSC）设置，并部分通过他们的模拟游戏（OP）算法解决，该算法在纸牌游戏Hanabi中显示了改进的ZSC和人类AI性能。OP假设可以访问环境的对称性，并防止代理在训练期间以相互不兼容的方式破坏这些对称性。然而，正如作者所指出的，发现给定环境的对称性是一个计算上的难题。相反，我们表明，通过简单的k级推理（KLR）Costa Gomes et al. 2006，同步训练所有级别，我们可以在Hanabi中获得有竞争力的ZSC和ad-hoc团队合作性能，包括与类人代理机器人配对时。我们还介绍了一种新的方法，同步k级推理与最佳响应（SyKLRBR），它进一步提高了我们的同步KLR的性能，通过共同训练的最佳响应。
Improving International Climate Policy via Mutually Conditional Binding Commitments
Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）；Alfred Weber Institute for Economics（阿尔弗雷德·韦伯经济研究所）；University of Heidelberg（海德堡大学）；Technical University of Berlin（柏林工业大学）；Dyson Robot Learning Lab（戴森机器人学习实验室）
