Object-centric representations have recently enabled significant progress in tackling relational reasoning tasks. By building a strong object-centric inductive bias into neural architectures, recent efforts have improved generalization and data efficiency of machine learning algorithms for these problems. One problem class involving relational reasoning that still remains under-explored is multi-agent reinforcement learning (MARL). Here we investigate whether object-centric representations are also beneficial in the fully cooperative MARL setting. Specifically, we study two ways of incorporating an agent-centric inductive bias into our RL algorithm: 1. Introducing an agent-centric attention module with explicit connections across agents 2. Adding an agent-centric unsupervised predictive objective (i.e. not using action labels), to be used as an auxiliary loss for MARL, or as the basis of a pre-training step. We evaluate these approaches on the Google Research Football environment as well as DeepMind Lab 2D. Empirically, agent-centric representation learning leads to the emergence of more complex cooperation strategies between agents as well as enhanced sample efficiency and generalization.
Deep Reinforcement Learning via Object-Centric Attention
Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；Sorbonne Université（索邦大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, Darmstadt, Germany（达姆施塔特认知科学中心）
Rutgers University（罗格斯大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）；Rutgers Center for Cognitive Science（罗格斯认知科学中心）
Object-centric world models provide structured representation of the scene and can be an important backbone in reinforcement learning and planning. However, existing approaches suffer in partially-observable environments due to the lack of belief states. In this paper, we propose Structured World Belief, a model for learning and inference of object-centric belief states. Inferred by Sequential Monte Carlo (SMC), our belief states provide multiple object-centric scene hypotheses. To synergize the benefits of SMC particles with object representations, we also propose a new object-centric dynamic…
Reinforcement learning agents must generalize beyond their training experience. Prior work has focused mostly on identical training and evaluation environments. Starting from the recently introduced Crafter benchmark, a 2D open world survival game, we introduce a new set of environments suitable for evaluating some agent's ability to generalize on previously unseen (numbers of) objects and to adapt quickly (meta-learning). In Crafter, the agents are evaluated by the number of unlocked achievements (such as collecting resources) when trained for 1M steps. We show that current agents struggle to…
Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；Centre for Cognitive Science of Darmstadt（达姆施塔特认知科学中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Solving complex real-world tasks, e.g., autonomous fleet control, often involves a coordinated team of multiple agents which learn strategies from visual inputs via reinforcement learning. Many existing multi-agent reinforcement learning (MARL) algorithms however don't scale to environments where agents operate on visual inputs. To address this issue, algorithmically, recent works have focused on non-stationarity and exploration. In contrast, we study whether scalability can also be achieved via a disentangled representation. For this, we explicitly construct an object-centric intermediate rep…
Towards Control-Centric Representations in Reinforcement Learning from Images
Beijing Institute of Electronic System Engineering（北京电子系统工程研究所）；Peking University（北京大学）；Beijing Institute of Technology（北京理工大学）；University of the Sunshine Coast（阳光海岸大学）
本文提出 ReBis，通过 Transformer 隐式建模动态并结合互模拟损失与非对称重构损失，在稀疏奖励下学习以控制为中心的图像强化学习表示，并在 Atari 和 DeepMind Control Suit 上验证了其优越性。
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）
Autonomous agents need large repertoires of skills to act reasonably on new tasks that they have not seen before. However, acquiring these skills using only a stream of high-dimensional, unstructured, and unlabeled observations is a tricky challenge for any autonomous agent. Previous methods have used variational autoencoders to encode a scene into a low-dimensional vector that can be used as a goal for an agent to discover new skills. Nevertheless, in compositional/multi-object environments it is difficult to disentangle all the factors of variation into such a fixed-length representation of …
Learning Interactive World Model for Object-Centric Reinforcement Learning
University of California San Diego（加州大学圣地亚哥分校）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫穆罕默德·本·扎耶德人工智能大学）；University of Amsterdam（阿姆斯特丹大学）
A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
Peking University（北京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Nankai University（南开大学）；Tongji University（同济大学）；Nanjing University（南京大学）；University of Wollongong（沃林根大学）；Shanghai Jiao Tong University（上海交通大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Human-Centric Aware UAV Trajectory Planning in Search and Rescue Missions Employing Multi-Objective Reinforcement Learning with AHP and Similarity-Based Experience Replay
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（卢森堡大学安全、可靠性与信任跨学科中心）；University of Luxembourg（卢森堡大学）
UTA Research Institute, University of Texas at Arlington（德克萨斯大学阿灵顿分校UTA研究所）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
The goal of this paper is to present a method for simultaneous trajectory and local stabilizing policy optimization to generate local policies for trajectory-centric model-based reinforcement learning (MBRL). This is motivated by the fact that global policy optimization for non-linear systems could be a very challenging problem both algorithmically and numerically. However, a lot of robotic manipulation tasks are trajectory-centric, and thus do not require a global model or policy. Due to inaccuracies in the learned model estimates, an open-loop trajectory optimization process mostly results i…
Interventions are central to causal learning and reasoning. Yet ultimately an intervention is an abstraction: an agent embedded in a physical environment (perhaps modeled as a Markov decision process) does not typically come equipped with the notion of an intervention -- its action space is typically ego-centric, without actions of the form `intervene on X'. Such a correspondence between ego-centric actions and interventions would be challenging to hard-code. It would instead be better if an agent learnt which sequence of actions allow it to make targeted manipulations of the environment, and …
Human-Centric Traffic Signal Control for Equity: A Multi-Agent Action Branching Deep Reinforcement Learning Approach
Department of Infrastructure Engineering, Faculty of Engineering and Information Technology, The University of Melbourne, VIC 3010, Australia（工程与信息技术学院基础设施工程系，墨尔本大学）
EDU-MATRIX: A Society-Centric Generative Cognitive Digital Twin Architecture for Secondary Education
The High School Affiliated to Beijing Normal University（北京师范大学附属高中）；Department of Electronic and Communication Engineering, North China Electric Power University（华北电力大学电子与通信工程系）
R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization
Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research Asia（微软亚洲研究院）；Hong Kong University of Science and Technology（香港科学与技术大学）；University of Oxford（牛津大学）
EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents
University of Alberta（阿尔伯塔大学）；National University of Singapore（新加坡国立大学）；Kuaishou Technology（快手科技）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）
Traditionally, the performance of multi-agent deep reinforcement learning algorithms are demonstrated and validated in gaming environments where we often have a fixed number of agents. In many industrial applications, the number of available agents can change at any given day and even when the number of agents is known ahead of time, it is common for an agent to break during the operation and become unavailable for a period of time. In this paper, we propose a new deep reinforcement learning algorithm for multi-agent collaborative tasks with a variable number of agents. We demonstrate the appl…
We examine the problem of adversarial reinforcement learning for multi-agent domains including a rule-based agent. Rule-based algorithms are required in safety-critical applications for them to work properly in a wide range of situations. Hence, every effort is made to find failure scenarios during the development phase. However, as the software becomes complicated, finding failure cases becomes difficult. Especially in multi-agent domains, such as autonomous driving environments, it is much harder to find useful failure scenarios that help us improve the algorithm. We propose a method for eff…
Nanjing University of Science and Technology（南京理工大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；Shanghai Jiao Tong University（上海交通大学）；Southeast University（东南大学）
The explosive growth of dynamic and heterogeneous data traffic brings great challenges for 5G and beyond mobile networks. To enhance the network capacity and reliability, we propose a learning-based dynamic time-frequency division duplexing (D-TFDD) scheme that adaptively allocates the uplink and downlink time-frequency resources of base stations (BSs) to meet the asymmetric and heterogeneous traffic demands while alleviating the inter-cell interference. We formulate the problem as a decentralized partially observable Markov decision process (Dec-POMDP) that maximizes the long-term expected su…
GRASP Laboratory（GRASP实验室）；University of Pennsylvania（宾夕法尼亚大学）
In this paper, we explore using deep reinforcement learning for problems with multiple agents. Most existing methods for deep multi-agent reinforcement learning consider only a small number of agents. When the number of agents increases, the dimensionality of the input and control spaces increase as well, and these methods do not scale well. To address this, we propose casting the multi-agent reinforcement learning problem as a distributed optimization problem. Our algorithm assumes that for multi-agent settings, policies of individual agents in a given population live close to each other in p…
Recent developments in deep reinforcement learning are concerned with creating decision-making agents which can perform well in various complex domains. A particular approach which has received increasing attention is multi-agent reinforcement learning, in which multiple agents learn concurrently to coordinate their actions. In such multi-agent environments, additional learning problems arise due to the continually changing decision-making policies of agents. This paper surveys recent works that address the non-stationarity problem in multi-agent deep reinforcement learning. The surveyed metho…
University of Illinois at Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
This article reviews recent advances in multi-agent reinforcement learning algorithms for large-scale control systems and communication networks, which learn to communicate and cooperate. We provide an overview of this emerging field, with an emphasis on the decentralized setting under different coordination protocols. We highlight the evolution of reinforcement learning algorithms from single-agent to multi-agent systems, from a distributed optimization perspective, and conclude with future directions and challenges, in the hope to catalyze the growing synergy among distributed optimization, …
We propose a general agent population learning system, and on this basis, we propose lineage evolution reinforcement learning algorithm. Lineage evolution reinforcement learning is a kind of derivative algorithm which accords with the general agent population learning system. We take the agents in DQN and its related variants as the basic agents in the population, and add the selection, mutation and crossover modules in the genetic algorithm to the reinforcement learning algorithm. In the process of agent evolution, we refer to the characteristics of natural genetic behavior, add lineage facto…
JetBrains Research（JetBrains研究院）；National Research University Higher School of Economics（国立研究型大学高等经济学院）；L3S Research Center（L3S研究中心）；Leibniz University Hannover（汉诺威莱布尼茨大学）
Over recent years, deep reinforcement learning has shown strong successes in complex single-agent tasks, and more recently this approach has also been applied to multi-agent domains. In this paper, we propose a novel approach, called MAGNet, to multi-agent reinforcement learning that utilizes a relevance graph representation of the environment obtained by a self-attention mechanism, and a message-generation technique. We applied our MAGnet approach to the synthetic predator-prey multi-agent environment and the Pommerman game and the results show that it significantly outperforms state-of-the-a…
Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）
How can we train a dialog model to produce better conversations by learning from human feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback from real-time, open-ended conversations, which we then use to train and improve the models using offline reinforcement learning (RL). We identify implicit conversational cues including language similarity, elicitation of laughter, sentiment, and more, which indicate positive human feedback, and embed these in multiple reward functions. A well-known challenge is that learning …
OCALM: Object-Centric Assessment with Language Models
LMU Munich（慕尼黑大学）；Munich Center of Machine Learning (MCML)（慕尼黑机器学习中心）；Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；National Research Center for Applied Cybersecurity (Athene)（国家应用网络安全研究中心）；Centre for Cognitive Science of Darmstadt（达姆施塔特认知科学中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Multi-agent reinforcement learning has shown promise on a variety of cooperative tasks as a consequence of recent developments in differentiable inter-agent communication. However, most architectures are limited to pools of homogeneous agents, limiting their applicability. Here we propose a modular framework for learning complex tasks in which a traditional monolithic agent is framed as a collection of cooperating heterogeneous agents. We apply this approach to model sensorimotor coordination in the neocortex as a multi-agent reinforcement learning problem. Our results demonstrate proof-of-con…
