Cooperative multi-agent tasks require agents to deduce their own contributions with shared global rewards, known as the challenge of credit assignment. General methods for policy based multi-agent reinforcement learning to solve the challenge introduce differentiate value functions or advantage functions for individual agents. In multi-agent system, polices of different agents need to be evaluated jointly. In order to update polices synchronously, such value functions or advantage functions also need synchronous evaluation. However, in current methods, value functions or advantage functions use counter-factual joint actions which are evaluated asynchronously, thus suffer from natural estimation bias. In this work, we propose the approximatively synchronous advantage estimation. We first derive the marginal advantage function, an expansion from single-agent advantage function to multi-agent system. Further more, we introduce a policy approximation for synchronous advantage estimation, and break down the multi-agent policy optimization problem into multiple sub-problems of single-agent policy optimization. Our method is compared with baseline algorithms on StarCraft multi-agent challenges, and shows the best performance on most of the tasks.
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
Mila Quebec AI Institute（魁北克AI研究所）；Université de Montréal（蒙特利尔大学）；Allen Institute for AI（人工智能研究院）；Google Deepmind（谷歌DeepMind）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Deep reinforcement learning (DRL) algorithms and evolution strategies (ES) have been applied to various tasks, showing excellent performances. These have the opposite properties, with DRL having good sample efficiency and poor stability, while ES being vice versa. Recently, there have been attempts to combine these algorithms, but these methods fully rely on synchronous update scheme, making it not ideal to maximize the benefits of the parallelism in ES. To solve this challenge, asynchronous update scheme was introduced, which is capable of good time-efficiency and diverse policy exploration. …
BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL
Beijing University of Posts and Telecommunications（北京邮电大学）；Baidu Inc.（百度公司）；Peking University（北京大学）；University of Science and Technology of China（中国科学技术大学）
Norwegian University of Science and Technology（挪威科技大学）；Telenor Research（Telenor研究）
We propose a novel framework for efficient parallelization of deep reinforcement learning algorithms, enabling these algorithms to learn from multiple actors on a single machine. The framework is algorithm agnostic and can be applied to on-policy, off-policy, value based and policy gradient based algorithms. Given its inherent parallelism, the framework can be efficiently implemented on a GPU, allowing the usage of powerful models while significantly reducing training time. We demonstrate the effectiveness of our framework by implementing an advantage actor-critic algorithm on a GPU, using on-…
Specialized Deep Residual Policy Safe Reinforcement Learning-Based Controller for Complex and Continuous State-Action Spaces
Software Competence Center Hagenberg, Data Science, Austria（哈根伯格软件中心数据科学部（奥地利））；Technological University Dublin, Computer Science, Dublin, Ireland（都柏林理工大学计算机科学系（爱尔兰都柏林））；ADAPT Research Centre, Dublin, Ireland（ADAPT研究中心（爱尔兰都柏林））
Laboratory for Information and Decision Systems (LIDS), MIT（麻省理工学院信息与决策系统实验室）；College of Computer and Information Science (CCIS), Northeastern University（东北大学计算机与信息科学学院）；Department of Electrical Engineering, Princeton University（普林斯顿大学电气工程系）
A key challenge in multi-robot and multi-agent systems is generating solutions that are robust to other self-interested or even adversarial parties who actively try to prevent the agents from achieving their goals. The practicality of existing works addressing this challenge is limited to only small-scale synchronous decision-making scenarios or a single agent planning its best response against a single adversary with fixed, procedurally characterized strategies. In contrast this paper considers a more realistic class of problems where a team of asynchronous agents with limited observation and…
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；JD Explore Academy（京东探索研究院）
Learning a risk-aware policy is essential but rather challenging in unstructured robotic tasks. Safe reinforcement learning methods open up new possibilities to tackle this problem. However, the conservative policy updates make it intractable to achieve sufficient exploration and desirable performance in complex, sample-expensive environments. In this paper, we propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. Concretely, the baseline …
Simons Institute for the Theory of Computing, UC Berkeley（加州大学伯克利分校西蒙斯计算理论研究所）；Departments of EECS and Statistics, UC Berkeley（加州大学伯克利分校电气工程与计算机科学系及统计学系）；Voleon Group, Berkeley（伯克利Voleon集团）
Markov reward processes (MRPs) are used to model stochastic phenomena arising in operations research, control engineering, robotics, and artificial intelligence, as well as communication and transportation networks. In many of these cases, such as in the policy evaluation problem encountered in reinforcement learning, the goal is to estimate the long-term value function of such a process without access to the underlying population transition and reward functions. Working with samples generated under the synchronous model, we study the problem of estimating the value function of an infinite-hor…
Georgia Institute of Technology（佐治亚理工学院）；Meta AI；Google Atlanta（谷歌亚特兰大）
We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). VER learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL). VER leads to significant and consistent speed-ups across a bro…
MARBLE: Multi-Armed Restless Bandits in Latent Markovian Environment
Department of Computer and Systems Sciences (DSV), Stockholm University（斯德哥尔摩大学计算机与系统科学系）；Inria Sophia Antipolis（法国国家信息与自动化研究所索菲亚·安蒂波利斯研究中心）；Smartprofile（Smartprofile公司）
RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Southern University of Science and Technology（南方科技大学）
University of California, San Diego（加州大学圣地亚哥分校）；San Diego State University（圣地亚哥州立大学）
This paper aims to mitigate straggler effects in synchronous distributed learning for multi-agent reinforcement learning (MARL) problems. Stragglers arise frequently in a distributed learning system, due to the existence of various system disturbances such as slow-downs or failures of compute nodes and communication bottlenecks. To resolve this issue, we propose a coded distributed learning framework, which speeds up the training of MARL algorithms in the presence of stragglers, while maintaining the same accuracy as the centralized approach. As an illustration, a coded distributed version of …
Google Research（谷歌研究院）；University of California, Berkeley（加州大学伯克利分校）；MIT（麻省理工学院）
Q-learning is a popular Reinforcement Learning (RL) algorithm which is widely used in practice with function approximation (Mnih et al., 2015). In contrast, existing theoretical results are pessimistic about Q-learning. For example, (Baird, 1995) shows that Q-learning does not converge even with linear function approximation for linear MDPs. Furthermore, even for tabular MDPs with synchronous updates, Q-learning was shown to have sub-optimal sample complexity (Li et al., 2021;Azar et al., 2013). The goal of this work is to bridge the gap between practical success of Q-learning and the relative…
Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization
Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Institute of Tsinghua University in Shenzhen（清华大学深圳研究院）
Khoury College of Computer Sciences（库利计算机科学学院）；Northeastern University（东北大学）
Synchronizing decisions across multiple agents in realistic settings is problematic since it requires agents to wait for other agents to terminate and communicate about termination reliably. Ideally, agents should learn and execute asynchronously instead. Such asynchronous methods also allow temporally extended actions that can take different amounts of time based on the situation and action executed. Unfortunately, current policy gradient methods are not applicable in asynchronous settings, as they assume that agents synchronously reason about action selection at every time step. To allow asy…
LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems
Division of Computer Science, The University of Hong Kong（计算机科学系，香港大学）；Department of Electrical and Electronic Engineering, The University of Hong Kong（电气电子工程系，香港大学）；Department of Computer Science and Technology, Peking University（计算机科学与技术系，北京大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，城市大学）；China Unicom Digital Technology, China Unicom co.,Ltd（中国联合数字技术，中国联合有限公司）；School of Computer and Information Engineering, Xiamen University of Technology（计算机与信息工程学院，厦门理工学院）；School of Computer Science, Engineering Research Center of Machine Learning and Industry Intelligence, Sichuan University（计算机科学学院，机器学习与工业智能工程研究中心，四川大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）；Institute of Space Internet, Fudan University（空间互联网研究院，复旦大学）；School of Computer Science, Fudan University（计算机科学学院，复旦大学）
The Ohio State University（俄亥俄州立大学）；National University of Singapore（新加坡国立大学）；Peng Cheng Laboratory（鹏城实验室）；Southern University of Science and Technology（南方科技大学）
Although Q-learning is one of the most successful algorithms for finding the best action-value function (and thus the optimal policy) in reinforcement learning, its implementation often suffers from large overestimation of Q-function values incurred by random sampling. The double Q-learning algorithm proposed in~\citet{hasselt2010double} overcomes such an overestimation issue by randomly switching the update between two Q-estimators, and has thus gained significant popularity in practice. However, the theoretical understanding of double Q-learning is rather limited. So far only the asymptotic …
Adaptive Value Decomposition: Coordinating a Varying Number of Agents in Urban Systems
State Key Laboratory of General Artificial Intelligence, BIGAI（人工智能通用基础理论国家重点实验室）；Beijing Technology and Business University（北京科技与商业大学）；Peking University（北京大学）
Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Electric motors are used in many applications and their efficiency is strongly dependent on their control. Among others, PI approaches or model predictive control methods are well-known in the scientific literature and industrial practice. A novel approach is to use reinforcement learning (RL) to have an agent learn electric drive control from scratch merely by interacting with a suitable control environment. RL achieved remarkable results with super-human performance in many games (e.g. Atari classics or Go) and also becomes more popular in control tasks like cartpole or swinging pendulum ben…
Princeton University（普林斯顿大学）；University of Pennsylvania（宾夕法尼亚大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）
Asynchronous Q-learning aims to learn the optimal action-value function (or Q-function) of a Markov decision process (MDP), based on a single trajectory of Markovian samples induced by a behavior policy. Focusing on a $γ$-discounted MDP with state space $\mathcal{S}$ and action space $\mathcal{A}$, we demonstrate that the $\ell_{\infty}$-based sample complexity of classical asynchronous Q-learning --- namely, the number of samples needed to yield an entrywise $\varepsilon$-accurate estimate of the Q-function --- is at most on the order of $\frac{1}{μ_{\min}(1-γ)^5\varepsilon^2}+ \frac{t_{mix}}…
