Many real-world scenarios involve teams of agents that have to coordinate their actions to reach a shared goal. We focus on the setting in which a team of agents faces an opponent in a zero-sum, imperfect-information game. Team members can coordinate their strategies before the beginning of the game, but are unable to communicate during the playing phase of the game. This is the case, for example, in Bridge, collusion in poker, and collusion in bidding. In this setting, model-free RL methods are oftentimes unable to capture coordination because agents' policies are executed in a decentralized fashion. Our first contribution is a game-theoretic centralized training regimen to effectively perform trajectory sampling so as to foster team coordination. When team members can observe each other actions, we show that this approach provably yields equilibrium strategies. Then, we introduce a signaling-based framework to represent team coordinated strategies given a buffer of past experiences. Each team member's policy is parametrized as a neural network whose output is conditioned on a suitable exogenous signal, drawn from a learned probability distribution. By combining these two elements, we empirically show convergence to coordinated equilibria in cases where previous state-of-the-art multi-agent RL algorithms did not.
Search and Rescue (SAR) missions in remote environments often employ autonomous multi-robot systems that learn, plan, and execute a combination of local single-robot control actions, group primitives, and global mission-oriented coordination and collaboration. Often, SAR coordination strategies are manually designed by human experts who can remotely control the multi-robot system and enable semi-autonomous operations. However, in remote environments where connectivity is limited and human intervention is often not possible, decentralized collaboration strategies are needed for fully-autonomous…
Many real-world problems require the coordination of multiple autonomous agents. Recent work has shown the promise of Graph Neural Networks (GNNs) to learn explicit communication strategies that enable complex multi-agent coordination. These works use models of cooperative multi-agent systems whereby agents strive to achieve a shared global goal. When considering agents with self-interested local objectives, the standard design choice is to model these as separate learning systems (albeit sharing the same environment). Such a design choice, however, precludes the existence of a single, differe…
Unsupervised skill discovery drives intelligent agents to explore the unknown environment without task-specific reward signal, and the agents acquire various skills which may be useful when the agents adapt to new tasks. In this paper, we propose "Multi-agent Skill Discovery"(MASD), a method for discovering skills for coordination patterns of multiple agents. The proposed method aims to maximize the mutual information between a latent code Z representing skills and the combination of the states of all agents. Meanwhile it suppresses the empowerment of Z on the state of any single agent by adve…
Recent studies in multi-agent communicative reinforcement learning (MACRL) have demonstrated that multi-agent coordination can be greatly improved by allowing communication between agents. Meanwhile, adversarial machine learning (ML) has shown that ML models are vulnerable to attacks. Despite the increasing concern about the robustness of ML algorithms, how to achieve robust communication in multi-agent reinforcement learning has been largely neglected. In this paper, we systematically explore the problem of adversarial communication in MACRL. Our main contributions are threefold. First, we pr…
With the evolution of various advanced driver assistance system (ADAS) platforms, the design of autonomous driving system is becoming more complex and safety-critical. The autonomous driving system simultaneously activates multiple ADAS functions; and thus it is essential to coordinate various ADAS functions. This paper proposes a randomized adversarial imitation learning (RAIL) method that imitates the coordination of autonomous vehicle equipped with advanced sensors. The RAIL policies are trained through derivative-free optimization for the decision maker that coordinates the proper ADAS fun…
School of Engineering, École Polytechnique Fédérale de Lausanne（瑞士联邦理工学院苏黎世校区工程学院）；Department of Electrical and Electronic Engineering, Imperial College London（伦敦帝国理工学院电子与电气工程系）
Tsinghua University（清华大学）；Louisiana State University（路易斯安那州立大学）
Learning to coordinate is a daunting problem in multi-agent reinforcement learning (MARL). Previous works have explored it from many facets, including cognition between agents, credit assignment, communication, expert demonstration, etc. However, less attention were paid to agents' decision structure and the hierarchy of coordination. In this paper, we explore the spatiotemporal structure of agents' decisions and consider the hierarchy of coordination from the perspective of multilevel emergence dynamics, based on which a novel approach, Learning to Advise and Learning from Advice (LALA), is p…
Evolving Interpretable Constitutions for Multi-Agent Coordination
College of Engineering Shibaura Institute of Technology（Shibaura Institute of Technology 工程学院）；Faculty of Arts and Sciences The University of Tokyo（东京大学 文理学部）；Department of EECS University of California, Berkeley（加州大学伯克利分校 电子工程与计算机科学系）；Department of Informatics Università degli Studi di Milano-Bicocca（米兰-比科卡大学 信息学系）
University of California, Irvine（加州大学欧文分校）；Carnegie Mellon University（卡内基梅隆大学）；Columbia University（哥伦比亚大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
Computing Nash equilibrium policies is a central problem in multi-agent reinforcement learning that has received extensive attention both in theory and in practice. However, provable guarantees have been thus far either limited to fully competitive or cooperative scenarios or impose strong assumptions that are difficult to meet in most practical applications. In this work, we depart from those prior results by investigating infinite-horizon \emph{adversarial team Markov games}, a natural and well-motivated class of games in which a team of identically-interested players -- in the absence of an…
Spatiotemporal Attention-Augmented Inverse Reinforcement Learning for Multi-Agent Task Allocation
College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统上海研究院）；Graz University of Technology（格拉茨技术大学）；Virtual Vehicle Research GmbH（虚拟车辆研究公司）
We enable efficient and effective coordination in unpredictable environments, i.e., in environments whose future evolution is unknown a priori and even adversarial. We are motivated by the future of autonomy that involves multiple robots coordinating in dynamic, unstructured, and adversarial environments to complete complex tasks such as target tracking, environmental mapping, and area monitoring. Such tasks are often modeled as submodular maximization coordination problems. We introduce the first submodular coordination algorithm with bounded tracking regret, i.e., with bounded suboptimality …
Area Protection in Adversarial Path-Finding Scenarios with Multiple Mobile Agents on Graphs: a theoretical and experimental study of target-allocation strategies for defense coordination
University of Bergen（卑尔根大学）；National Institute of Advanced Industrial Science and Technology (AIST)（产业技术综合研究所）
Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
Mila, Quebec Artificial Intelligence Institute, Université de Montréal（蒙特利尔大学人工智能研究所，魁北克机器学习研究院）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）；Massachusetts Institute of Technology（麻省理工学院）；Georgia Institute of Technology（佐治亚理工学院）；University of Washington（华盛顿大学）；McGill University（麦吉尔大学）；Independent Researcher（独立研究者）
Guidelines for Applying RL and MARL in Cybersecurity Applications
Alan Turing Institute（阿兰·图灵研究所）；BAE Systems（BAE系统公司）；Defence Science and Technology Laboratory（国防科学技术实验室）；BMT Group（BMT集团）；Applied Data Science Partners（应用数据科学合作伙伴公司）；Montvieux（蒙维尤公司）；Frazer-Nash Consultancy（弗雷泽-纳什咨询公司）
Many tasks in practice require the collaboration of multiple agents through reinforcement learning. In general, cooperative multiagent reinforcement learning algorithms can be classified into two paradigms: Joint Action Learners (JALs) and Independent Learners (ILs). In many practical applications, agents are unable to observe other agents' actions and rewards, making JALs inapplicable. In this work, we focus on independent learning paradigm in which each agent makes decisions based on its local observations only. However, learning is challenging in independent settings due to the local viewpo…
Adaptive networks have the capability to pursue solutions of global stochastic optimization problems by relying only on local interactions within neighborhoods. The diffusion of information through repeated interactions allows for globally optimal behavior, without the need for central coordination. Most existing strategies are developed for cooperative learning settings, where the objective of the network is common to all agents. We consider in this work a team setting, where a subset of the agents form a team with a common goal while competing with the remainder of the network. We develop an…
