Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games
Vrije Universiteit Brussel（布鲁塞尔自由大学）；Universiteit van Amsterdam（阿姆斯特丹大学）；National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）；HU Univ. of Appl. Sci. Utrecht（乌得勒支HU应用科学大学）
Roxana Rădulescu, Timothy Verstraeten, Yijie Zhang, Patrick Mannion, Diederik M. Roijers, Ann Nowé
Many real-world multi-agent interactions consider multiple distinct criteria, i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. In this work, we present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with non-linear utilities. Specifically, we consider two-player multi-objective normal form games with non-linear utility functions under the scalarised expected returns optimisation criterion. We contribute novel actor-critic and policy gradient formulations to allow reinforcement learning of mixed strategies in this setting, along with extensions that incorporate opponent policy reconstruction and learning with opponent learning awareness (i.e., learning while considering the impact of one's policy when anticipating the opponent's learning step). Empirical results in five different MONFGs demonstrate that opponent learning awareness and modelling can drastically alter the learning dynamics in this setting. When equilibria are present, opponent modelling can confer significant benefits on agents that implement it. When there are no Nash equilibria, opponent learning awareness and modelling allows agents to still converge to meaningful solutions that approximate equilibria.
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of Oxford（牛津大学）；FLAIR（未来人工智能研究院）
Learning With Opponent-Learning Awareness (LOLA) (Foerster et al. [2018a]) is a multi-agent reinforcement learning algorithm that typically learns reciprocity-based cooperation in partially competitive environments. However, LOLA often fails to learn such behaviour on more complex policy spaces parameterized by neural networks, partly because the update rule is sensitive to the policy parameterization. This problem is especially pronounced in the opponent modeling setting, where the opponent's policy is unknown and must be inferred from observations; in such settings, LOLA is ill-specified bec…
University of Oxford（牛津大学）；OpenAI；Carnegie Mellon University（卡内基梅隆大学）；UC Berkeley（加州大学伯克利分校）
Multi-agent settings are quickly gathering importance in machine learning. This includes a plethora of recent work on deep multi-agent reinforcement learning, but also can be extended to hierarchical RL, generative adversarial networks and decentralised optimisation. In all these settings the presence of multiple learning agents renders the training problem non-stationary and often leads to unstable training or undesired final results. We present Learning with Opponent-Learning Awareness (LOLA), a method in which each agent shapes the anticipated learning of the other agents in the environment…
Evolving in the Agent Jungle via History-Informed Opponent Awareness
University of Edinburgh（爱丁堡大学）；University of Oklahoma（俄克拉荷马大学）；Imperial College London（伦敦帝国学院）；University of Michigan（密歇根大学）；University of Southern California（南加州大学）；Tencent（腾讯）；Beijing Normal University（北京师范大学）；Beijing Normal–Hong Kong Baptist University（北京师范大学-香港浸会大学联合国际学院）
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
University of Oxford（牛津大学）；DeepMind（深度思维）；University College London（伦敦大学学院）
A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -- from GANs and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player influence on others' updates. Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner's Dilemma. Although experimentally successful, we show that LOLA agents can exhibit 'arrogant' behavio…
Multi-Agent Reinforcement Learning (MARL) considers settings in which a set of coexisting agents interact with one another and their environment. The adaptation and learning of other agents induces non-stationarity in the environment dynamics. This poses a great challenge for value function-based algorithms whose convergence usually relies on the assumption of a stationary environment. Policy search algorithms also struggle in multi-agent settings as the partial observability resulting from an opponent's actions not being known introduces high variance to policy training. Modelling an agent's …
We propose Searching with Opponent-Awareness (SOA), an approach to leverage opponent-aware planning without explicit or a priori opponent models for improving performance and social welfare in multi-agent systems. To this end, we develop an opponent-aware MCTS scheme using multi-armed bandits based on Learning with Opponent-Learning Awareness (LOLA) and compare its effectiveness with other bandits, including UCB1. Our evaluations include several different settings and show the benefits of SOA are especially evident with increasing number of agents.
School of Computer and Information Science, the Southwest University（西南大学计算机与信息科学学院）；University of Liverpool（利物浦大学）；School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；School of Computer Software, Tianjin University（天津大学计算机软件学院）
In multiagent environments, the capability of learning is important for an agent to behave appropriately in face of unknown opponents and dynamic environment. From the system designer's perspective, it is desirable if the agents can learn to coordinate towards socially optimal outcomes, while also avoiding being exploited by selfish opponents. To this end, we propose a novel gradient ascent based algorithm (SA-IGA) which augments the basic gradient-ascent algorithm by incorporating social awareness into the policy update process. We theoretically analyze the learning dynamics of SA-IGA using d…
University College London（伦敦大学学院）；Huawei Research & Development U.K.（华为英国研发中心）
Though limited in real-world decision making, most multi-agent reinforcement learning (MARL) models assume perfectly rational agents -- a property hardly met due to individual's cognitive limitation and/or the tractability of the decision problem. In this paper, we introduce generalized recursive reasoning (GR2) as a novel framework to model agents with different \emph{hierarchical} levels of rationality; our framework enables agents to exhibit varying levels of "thinking" ability thereby allowing higher-level agents to best respond to various less sophisticated learners. We contribute both th…
Mathematics of statistical sequential decision-making: concentration, risk-awareness and modelling in stochastic bandits, with applications to bariatric surgery
Université de Lille（里尔大学）；Université de Technologies de Troyes（特鲁瓦技术大学）；ENSAI Rennes（雷恩国立统计与信息分析学校）；Inria Montpellier（法国国家信息与自动化研究所蒙彼利埃中心）；CWI Amsterdam（荷兰国家数学与计算机科学研究中心）；Université de Bordeaux（波尔多大学）；Inria Lille（法国国家信息与自动化研究所里尔中心）；CHU Lille（里尔大学附属医院）；Centre de Recherche en Informatique, Signal et Automatique de Lille（里尔计算机、信号与自动化研究中心）
Tsinghua University（清华大学）；Hasso Plattner Institute（哈索·普拉特纳研究所）；University of Potsdam（波茨坦大学）
Emergentism and pragmatics are two research fields that study the dynamics of linguistic communication along substantially different timescales and intelligence levels. From the perspective of multi-agent reinforcement learning, they correspond to stochastic games with reinforcement training and stage games with opponent awareness. Given that their combination has been explored in linguistics, we propose computational models that combine short-term mutual reasoning-based pragmatics with long-term language emergentism. We explore this for agent communication referential games as well as in Star…
This paper surveys the field of deep multiagent reinforcement learning. The combination of deep neural networks with reinforcement learning has gained increased traction in recent years and is slowly shifting the focus from single-agent to multiagent environments. Dealing with multiple agents is inherently more complex as (a) the future rewards depend on multiple players' joint actions and (b) the computational complexity increases. We present the most common multiagent problem representations and their main challenges, and identify five research areas that address one or more of these challen…
In general-sum games, the interaction of self-interested learning agents commonly leads to collectively worst-case outcomes, such as defect-defect in the iterated prisoner's dilemma (IPD). To overcome this, some methods, such as Learning with Opponent-Learning Awareness (LOLA), shape their opponents' learning process. However, these methods are myopic since only a small number of steps can be anticipated, are asymmetric since they treat other agents as naive learners, and require the use of higher-order derivatives, which are calculated through white-box access to an opponent's differentiable …
JPMorgan Chase & Co.（摩根大通公司）；University College London（伦敦大学学院）；University of Oxford（牛津大学）
Multi-agent learning is a promising method to simulate aggregate competitive behaviour in finance. Learning expert agents' reward functions through their external demonstrations is hence particularly relevant for subsequent design of realistic agent-based simulations. Inverse Reinforcement Learning (IRL) aims at acquiring such reward functions through inference, allowing to generalize the resulting policy to states not observed in the past. This paper investigates whether IRL can infer such rewards from agents within real financial stochastic environments: limit order books (LOB). We introduce…
Cooperation is a phenomenon that has been widely studied across many different disciplines. In the field of computer science, the modularity and robustness of multi-agent systems offer significant practical advantages over individual machines. At the same time, agents using standard reinforcement learning algorithms often fail to achieve long-term, cooperative strategies in unstable environments when there are short-term incentives to defect. Political philosophy, on the other hand, studies the evolution of cooperation in humans who face similar incentives to act individualistically, but never…
Replacing traditional fossil fuel vehicles with innovative zero-emission vehicles for the transport in ci ties is one of the major tactics to achieve the UK government 2020 target of cutting emission. We are developing an agent-based simulation model to study the possible impact of different governmental interventions on the diffusion of such vehicles. Options that could be studied with our what-if analysis to include things like car parking charges, price of electrical car, energy awareness and word of mouth. In this paper we present a first case study related to the introduction of a new car…
