Being able to predict the mental states of others is a key factor to effective social interaction. It is also crucial for distributed multi-agent systems, where agents are required to communicate and cooperate. In this paper, we introduce such an important social-cognitive skill, i.e. Theory of Mind (ToM), to build socially intelligent agents who are able to communicate and cooperate effectively to accomplish challenging tasks. With ToM, each agent is capable of inferring the mental states and intentions of others according to its (local) observation. Based on the inferred states, the agents decide "when" and with "whom" to share their intentions. With the information observed, inferred, and received, the agents decide their sub-goals and reach a consensus among the team. In the end, the low-level executors independently take primitive actions to accomplish the sub-goals. We demonstrate the idea in two typical target-oriented multi-agent tasks: cooperative navigation and multi-sensor target coverage. The experiments show that the proposed model not only outperforms the state-of-the-art methods on reward and communication efficiency, but also shows good generalization across different scales of the environment.
Improving Multi-Agent Cooperation using Theory of Mind
National University of Singapore（新加坡国立大学）；AI Singapore（新加坡人工智能计划）；Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究院）
Emergence of Theory of Mind Collaboration in Multiagent Systems
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
目前，在多智能体系统的研究中，往往忽略了智能体的意图。尽管如此，正如心理理论（Theory of Mind，ToM）所指出的，人们经常会对他人的心理状态进行推理，包括信念、目标和意图，以在竞争、合作或联盟中获得绩效优势。然而，由于其固有的递归性和难以建模的分布信念，集成ToM的多智能体规划和决策仍然是一个挑战。在本文中，我们将ToM在多智能体部分可观察马尔可夫决策过程（POMDP），并提出了一个自适应训练算法，开发有效的合作与ToM代理。我们用两个游戏来评估我们的算法，我们的算法超过了以前所有的分散执行算法，而没有对ToM进行建模。
UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios
多智能体强化学习方法如VDN、QMIX和QTRAN采用集中训练与分散执行（CTDE）框架，在合作和竞争方面显示出良好的效果。然而，在一些多代理场景中，代理的数量和动作集的大小实际上随时间而变化。我们称之为不成形的场景，上面提到的方法都不能令人满意地执行。在本文中，我们提出了一种新的方法称为Unshaped Networks for Multi-Agent Systems（UNMAS），以适应多智能体系统的数量和大小的变化。我们提出了自加权混合网络来分解联合作用值。它对个体数量变化的适应性归因于个体Q值到个体权重的联合动作值的非线性映射。此外，为了解决行动集的变化，每个代理构建一个单独的行动价值网络，该网络由两个流组成，以评估恒定的环境导向子集和变化的单元导向子集。我们评估UNMAS在各种星际争霸II微观管理方案，并比较结果与几个国家的最先进的MARL算法。地雷行动处的优越性体现在其最高的中奖率上，特别是在最困难的3s5z_vs_3s6z情况下。当其他MARL算法失败时，智能体学习执行有效的合作行为。https://sites.google.com/view/unmas提供了动画演示和源代码。
