We prove under commonly used assumptions the convergence of actor-critic reinforcement learning algorithms, which simultaneously learn a policy function, the actor, and a value function, the critic. Both functions can be deep neural networks of arbitrary complexity. Our framework allows showing convergence of the well known Proximal Policy Optimization (PPO) and of the recently introduced RUDDER. For the convergence proof we employ recently introduced techniques from the two time-scale stochastic approximation theory. Our results are valid for actor-critic methods that use episodic samples and that have a policy that becomes more greedy during learning. Previous convergence proofs assume linear function approximation, cannot treat episodic examples, or do not consider that policies become greedy. The latter is relevant since optimal policies are typically deterministic.
Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）；Institute for Machine Learning（机器学习研究所）；LIT AI Lab（LIT人工智能实验室）；Institute of Advanced Research in Artificial Intelligence (IARAI)（人工智能高级研究所（IARAI））
We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs). In MDPs the Q-values are equal to the expected immediate reward plus the expected future rewards. The latter are related to bias problems in temporal difference (TD) learning and to high variance problems in Monte Carlo (MC) learning. Both problems are even more severe when rewards are delayed. RUDDER aims at making the expected future rewards zero, which simplifies Q-value estimation to computing the mean of the immediate reward. We propose the following two new concepts …
Norwegian University of Science and Technology（挪威科技大学）
Previous studies on automatic berthing systems based on artificial neural network (ANN) showed great berthing performance by training the ANN with ship berthing data as training data. However, because the ANN requires a large amount of training data to yield robust performance, the ANN-based automatic berthing system is somewhat limited due to the difficulty in obtaining the berthing data. In this study, to overcome this difficulty, the automatic berthing system based on one of the reinforcement learning (RL) algorithms, proximal policy optimization (PPO), is proposed because the RL algorithms…
Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Westlake University（西湖大学）；Nanjing University（南京大学）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-actor networks. While this paper is mainly focused on parameterized action space, the proposed architecture, which we call hybrid actor-critic, can be extended for more general action spaces which has a hierarchical structure. We present an instance of the hybrid actor-critic archite…
Start Classifying: Categorical Critics for LLM Reinforcement Learning
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；Tencent YoutuLab（腾讯优图实验室）；Griffith University（格里菲斯大学）；Shanghai Academy of Artificial Intelligence for Science（上海科学人工智能研究院）
Drexel University（德雷塞尔大学）；University of Eastern Finland（东芬兰大学）；German Aerospace Center（德国航空航天中心）；Tianjin University（天津大学）；Kobe University（神户大学）
Advantage Actor-critic (A2C) and Proximal Policy Optimization (PPO) are popular deep reinforcement learning algorithms used for game AI in recent years. A common understanding is that A2C and PPO are separate algorithms because PPO's clipped objective appears significantly different than A2C's objective. In this paper, however, we show A2C is a special case of PPO. We present theoretical justifications and pseudocode analysis to demonstrate why. To validate our claim, we conduct an empirical experiment using \texttt{Stable-baselines3}, showing A2C and PPO produce the \textit{exact} same models…
Tsinghua University（清华大学）；University of California, Berkeley（加州大学伯克利分校）；Shanghai Qi Zhi Institute（上海期智研究院）
Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due to the belief that PPO is significantly less sample efficient than off-policy methods in multi-agent systems. In this work, we carefully study the performance of PPO in cooperative multi-agent settings. We show that PPO-based multi-agent algorithms achieve surprisingly strong performance in four popular multi-agent testbeds: the particle-world environments, the StarCraft multi-agent challe…
Austrian Institute of Technology（奥地利技术研究所）；FH Joanneum（约翰内姆应用科技大学）；Google Research, Brain Team（谷歌研究院大脑团队）
Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are naturally on-policy. In this paper, we propose to combine (any kind of) soft greediness with Modified Policy Iteration (MPI). The proposed abstract framework applies repeatedly: (i) a partial policy evaluation step that allows off-policy learning and (ii) any softened greedy ste…
In this paper, a novel racing environment for OpenAI Gym is introduced. This environment operates with continuous action- and state-spaces and requires agents to learn to control the acceleration and steering of a car while navigating a randomly generated racetrack. Different versions of two actor-critic learning algorithms are tested on this environment: Sampled Policy Gradient (SPG) and Proximal Policy Optimization (PPO). An extension of SPG is introduced that aims to improve learning performance by weighting action samples during the policy update step. The effect of using experience replay…
Proximal policy optimization (PPO) has yielded state-of-the-art results in policy search, a subfield of reinforcement learning, with one of its key points being the use of a surrogate objective function to restrict the step size at each policy update. Although such restriction is helpful, the algorithm still suffers from performance instability and optimization inefficiency from the sudden flattening of the curve. To address this issue we present a PPO variant, named Proximal Policy Optimization Smooth Algorithm (PPOS), and its critical improvement is the use of a functional clipping method in…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is still far from being fully understood. In this paper, we show that PPO could neither strictly restrict the likelihood ratio as it attempts to do nor enforce a well-defined trust region constraint, which means that it may still suffer from the risk of performance instability. To address this issue, we present an enhanced PPO method, named Truly PPO. Two critical improvements ar…
A Reinforcement Learning Method for Environments with Stochastic Variables: Post-Decision Proximal Policy Optimization with Dual Critic Networks
Division of Computer Science Instituto Tecnológico de Aeronáutica（计算机科学系 航空航天技术研究院）；Department of Mathematical Sciences Politecnico di Torino（数学科学系 波尔蒂纳罗内理工大学）；Escola Politécnica Universidade de São Paulo（工学院 圣保罗大学）；Instituto Tecnológico de Aeronáutica（航空航天技术研究院）
Learning with a Single Rollout via Pass@k Policy Optimization
University of Alberta（阿尔伯塔大学）；BIGAI（北京通用人工智能研究院）；University of Toronto（多伦多大学）；McGill University, Mila（麦吉尔大学，米拉）；Nvidia Research（英伟达研究院）；Amii（阿尔伯塔机器智能研究所）；CIFAR AI Chair（CIFAR人工智能教席）
Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to nonconvexity, the global convergence of PPO and TRPO remains less understood, which separates theory from practice. In this paper, we prove that a variant of PPO and TRPO equipped with overparametrized neural networks converges to the globally optimal policy at a sublinear rate. The key to our analysis is the global convergence of infinite-dimensional mirror descent under a no…
Sandia National Laboratories（桑迪亚国家实验室）；University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）
Vision-based deep reinforcement learning (RL) typically obtains performance benefit by using high capacity and relatively large convolutional neural networks (CNN). However, a large network leads to higher inference costs (power, latency, silicon area, MAC count). Many inference optimizations have been developed for CNNs. Some optimization techniques offer theoretical efficiency, such as sparsity, but designing actual hardware to support them is difficult. On the other hand, distillation is a simple general-purpose optimization technique which is broadly applicable for transferring knowledge f…
Fault-tolerant flight control faces challenges, as developing a model-based controller for each unexpected failure is unrealistic, and online learning methods can handle limited system complexity due to their low sample efficiency. In this research, a model-free coupled-dynamics flight controller for a jet aircraft able to withstand multiple failure types is proposed. An offline trained cascaded Soft Actor-Critic Deep Reinforcement Learning controller is successful on highly coupled maneuvers, including a coordinated 40 degree bank climbing turn with a normalized Mean Absolute Error of 2.64%. …
University of Manchester（曼彻斯特大学）；University of Oxford（牛津大学）；Microsoft Research（微软研究院）
We present trust region bounds for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary. This new analysis provides a theoretical understanding of the strong performance of two recent actor-critic methods for MARL, which both rely on independent ratios, i.e., computing probability ratios separately for each agent's policy. We show that, despite the non-stationarity that independent ratios cause, a monotonic improvement guarantee still arises as a result of enforcing the trust region constrai…
Georgia Institute of Technology（佐治亚理工学院）；Northwestern University（西北大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Multi-agent reinforcement learning (MARL) becomes more challenging in the presence of more agents, as the capacity of the joint state and action spaces grows exponentially in the number of agents. To address such a challenge of scale, we identify a class of cooperative MARL problems with permutation invariance, and formulate it as a mean-field Markov decision processes (MDP). To exploit the permutation invariance therein, we propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation-invariant actor-critic neural architecture. We prove that MF-…
