We consider the problem of robust multi-agent reinforcement learning (MARL) for cooperative communication and coordination tasks. MARL agents, mainly those trained in a centralized way, can be brittle because they can adopt policies that act under the expectation that other agents will act a certain way rather than react to their actions. Our objective is to bias the learning process towards finding strategies that remain reactive towards others' behavior. Social empowerment measures the potential influence between agents' actions. We propose it as an additional reward term, so agents better adapt to other agents' actions. We show that the proposed method results in obtaining higher rewards faster and a higher success rate in three cooperative communication and coordination tasks.
BMW Group（宝马集团）；University of Amsterdam（阿姆斯特丹大学）；University of Hertfordshire（赫特福德大学）
We consider multi-agent reinforcement learning (MARL) for cooperative communication and coordination tasks. MARL agents can be brittle because they can overfit their training partners' policies. This overfitting can produce agents that adopt policies that act under the expectation that other agents will act in a certain way rather than react to their actions. Our objective is to bias the learning process towards finding reactive strategies towards other agents' behaviors. Our method, transfer empowerment, measures the potential influence between agents' actions. Results from three simulated co…
Mobile robot navigation has seen extensive research in the last decades. The aspect of collaboration with robots and humans sharing workspaces will become increasingly important in the future. Therefore, the next generation of mobile robots needs to be socially-compliant to be accepted by their human collaborators. However, a formal definition of compliance is not straightforward. On the other hand, empowerment has been used by artificial agents to learn complicated and generalized actions and also has been shown to be a good model for biological behaviors. In this paper, we go beyond the appr…
Unsupervised skill discovery drives intelligent agents to explore the unknown environment without task-specific reward signal, and the agents acquire various skills which may be useful when the agents adapt to new tasks. In this paper, we propose "Multi-agent Skill Discovery"(MASD), a method for discovering skills for coordination patterns of multiple agents. The proposed method aims to maximize the mutual information between a latent code Z representing skills and the combination of the states of all agents. Meanwhile it suppresses the empowerment of Z on the state of any single agent by adve…
Joint attention - the ability to purposefully coordinate attention with another agent, and mutually attend to the same thing -- is a critical component of human social cognition. In this paper, we ask whether joint attention can be useful as a mechanism for improving multi-agent coordination and social learning. We first develop deep reinforcement learning (RL) agents with a recurrent visual attention architecture. We then train agents to minimize the difference between the attention weights that they apply to the environment at each timestep, and the attention of other agents. Our results sho…
This note confirms a conjecture of [Bramoullé, Anti-coordination and social interactions, Games and Economic Behavior, 58, 2007: 30-49]. The problem, which we name the maximum independent cut problem, is a restricted version of the MAX-CUT problem, requiring one side of the cut to be an independent set. We show that the maximum independent cut problem does not admit any polynomial time algorithm with approximation ratio better than $n^{1-ε}$, where $n$ is the number of nodes, and $ε$ arbitrarily small, unless P=NP. For the rather special case where each node has a degree of at most four, the p…
It has been argued that one role of social constructs, such as institutions, trust and norms, is to coordinate the expectations of autonomous entities in order to resolve collective action situations (such as collective risk dilemmas) through the coordination of behaviour. While much work has addressed the formal representation of these social constructs, in this paper we focus specifically on the formal representation of, and associated reasoning with, the expectations themselves. In particular, we investigate how explicit reasoning about expectations can be used to encode both traditional ga…
Evolving Interpretable Constitutions for Multi-Agent Coordination
College of Engineering Shibaura Institute of Technology（Shibaura Institute of Technology 工程学院）；Faculty of Arts and Sciences The University of Tokyo（东京大学 文理学部）；Department of EECS University of California, Berkeley（加州大学伯克利分校 电子工程与计算机科学系）；Department of Informatics Università degli Studi di Milano-Bicocca（米兰-比科卡大学 信息学系）
Multiagent coordination in cooperative multiagent systems (MASs) has been widely studied in both fixed-agent repeated interaction setting and the static social learning framework. However, two aspects of dynamics in real-world multiagent scenarios are currently missing in existing works. First, the network topologies can be dynamic where agents may change their connections through rewiring during the course of interactions. Second, the game matrix between each pair of agents may not be static and usually not known as a prior. Both the network dynamic and game uncertainty increase the coordinat…
In this paper, we introduce a new class of potential fields, i.e., meta navigation functions (MNFs) to coordinate multi-agent systems. Thanks to the MNF formulation, agents can contribute to each other's coordination via partial and/or total associations, contrary to traditional decentralized navigation functions (DNFs). In particular, agents may stimulate each other via their MNFs. Moreover, MNFs need to be confined which is a weaker condition compared to the Morse condition of DNFs. An MNF is composed of a confined function and an attraction kernel. The critical points of the former can be c…
We address two major challenges of implicit coordination in multi-agent deep reinforcement learning: non-stationarity and exponential growth of state-action space, by combining Deep-Q Networks for policy learning with Nash equilibrium for action selection. Q-values proxy as payoffs in Nash settings, and mutual best responses define joint action selection. Coordination is implicit because multiple/no Nash equilibria are resolved deterministically. We demonstrate that knowledge of game type leads to an assumption of mirrored best responses and faster convergence than Nash-Q. Specifically, the Fr…
Massachusetts Institute of Technology（麻省理工学院）；Google DeepMind（谷歌DeepMind）；Princeton University（普林斯顿大学）
We propose a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning (MARL), through rewarding agents for having causal influence over other agents' actions. Causal influence is assessed using counterfactual reasoning. At each timestep, an agent simulates alternate actions that it could have taken, and computes their effect on the behavior of other agents. Actions that lead to bigger changes in other agents' behavior are considered influential and are rewarded. We show that this is equivalent to rewarding agents for having high mutual information be…
Department of Computer Science, Bar-Ilan University（巴伊兰大学计算机科学系）；Department of Computer Science, University of Oxford（牛津大学计算机科学系）；Institute for Decentralized AI (IDAI)（去中心化人工智能研究所）
Differential Inequalities in Multi-Agent Coordination and Opinion Dynamics Modeling
Delft University of Technology（代尔夫特理工大学）；Institute for Problems of Mechanical Engineering, Russian Academy of Sciences（俄罗斯科学院机械问题研究所）；ITMO University（ITMO大学）；University of Groningen（格罗宁根大学）
Upon starting a collective endeavour, it is important to understand your partners' preferences and how strongly they commit to a common goal. Establishing a prior commitment or agreement in terms of posterior benefits and consequences from those engaging in it provides an important mechanism for securing cooperation. Resorting to methods from Evolutionary Game Theory (EGT), here we analyse how prior commitments can also be adopted as a tool for enhancing coordination when its outcomes exhibit an asymmetric payoff structure, in both pairwise and multiparty interactions. Arguably, coordination i…
