This paper is the arXiv version with Appendices of the published AAAI paper: "Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach," in Proc. AAAI, Feb 2022. The paper has been further extended with concave utilities and constraints in v2
Reinforcement learning is widely used in applications where one needs to perform sequential decisions while interacting with the environment. The problem becomes more challenging when the decision requirement includes satisfying some safety constraints. The problem is mathematically formulated as constrained Markov decision process (CMDP). In the literature, various algorithms are available to solve CMDP problems in a model-free manner to achieve $ε$-optimal cumulative reward with $ε$ feasible policies. An $ε$-feasible policy implies that it suffers from constraint violation. An important question here is whether we can achieve $ε$-optimal cumulative reward with zero constraint violations or not. To achieve that, we advocate the use of randomized primal-dual approach to solve the CMDP problems and propose a conservative stochastic primal-dual algorithm (CSPDA) which is shown to exhibit $\tilde{\mathcal{O}}\left(1/ε^2\right)$ sample complexity to achieve $ε$-optimal cumulative reward with zero constraint violations. In the prior works, the best available sample complexity for the $ε$-optimal policy with zero constraint violation is $\tilde{\mathcal{O}}\left(1/ε^5\right)$. Hence, the proposed algorithm provides a significant improvement as compared to the state of the art.
We address the issue of safety in reinforcement learning. We pose the problem in an episodic framework of a constrained Markov decision process. Existing results have shown that it is possible to achieve a reward regret of $\tilde{\mathcal{O}}(\sqrt{K})$ while allowing an $\tilde{\mathcal{O}}(\sqrt{K})$ constraint violation in $K$ episodes. A critical question that arises is whether it is possible to keep the constraint violation even smaller. We show that when a strictly safe policy is known, then one can confine the system to zero constraint violation with arbitrarily high probability while …
We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint vi…
Near-Constant Strong Violation and Last-Iterate Convergence for Online CMDPs via Decaying Safety Margins
School of Informatics, University of Edinburgh, Edinburgh, United Kingdom（爱丁堡大学信息学院）；School of Mathematics, University of Edinburgh, Edinburgh, United Kingdom（爱丁堡大学数学学院）
A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees
The University of Tokyo（东京大学）；OMRON SINIC X（欧姆龙SINIC X公司）；Mizuho–DL Financial Technology（瑞穗-DL金融科技公司）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所革新智能综合研究中心）；Osaka University（大阪大学）；Kyoto University（京都大学）
We address the problem of finding the optimal policy of a constrained Markov decision process (CMDP) using a gradient descent-based algorithm. Previous results have shown that a primal-dual approach can achieve an $\mathcal{O}(1/\sqrt{T})$ global convergence rate for both the optimality gap and the constraint violation. We propose a new algorithm called policy mirror descent-primal dual (PMD-PD) algorithm that can provably achieve a faster $\mathcal{O}(\log(T)/T)$ convergence rate for both the optimality gap and the constraint violation. For the primal (policy) update, the PMD-PD algorithm uti…
Primal-Dual Policy Optimization for Linear CMDPs with Adversarial Losses
Research Institute of Mathematics, Seoul National University（数学研究所，首尔国立大学）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能跨学科项目，首尔国立大学）；Korea Institute for Advanced Study（韩国高级研究院）
Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes
School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电气与计算机工程学院）；Bradley Department of Electrical and Computer Engineering, Virginia Tech（弗吉尼亚理工大学布拉德利电气与计算机工程学院）
The Ohio State University（俄亥俄州立大学）；Georgia Institute of Technology（佐治亚理工学院）
In safe reinforcement learning (SRL) problems, an agent explores the environment to maximize an expected total reward and meanwhile avoids violation of certain constraints on a number of expected total costs. In general, such SRL problems have nonconvex objective functions subject to multiple nonconvex constraints, and hence are very challenging to solve, particularly to provide a globally optimal policy. Many popular SRL algorithms adopt a primal-dual structure which utilizes the updating of dual variables for satisfying the constraints. In contrast, we propose a primal approach, called const…
Simon Fraser University（西蒙弗雷泽大学）；SAIT AI Lab（SAIT人工智能实验室）；Amii（阿尔伯塔机器智能研究所）；University of Alberta（阿尔伯塔大学）；DeepMind（深度思维）；Mila（米拉人工智能研究所）；McGill University（麦吉尔大学）
We study policy optimization in an infinite horizon, $γ$-discounted constrained Markov decision process (CMDP). Our objective is to return a policy that achieves large expected reward with a small constraint violation. We consider the online setting with linear function approximation and assume global access to the corresponding features. We propose a generic primal-dual framework that allows us to bound the reward sub-optimality and constraint violation for arbitrary algorithms in terms of their primal and dual regret on online linear optimization problems. We instantiate this framework to us…
Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Beijing International Center for Mathematical Research（北京国际数学研究中心）
As an important framework for safe Reinforcement Learning, the Constrained Markov Decision Process (CMDP) has been extensively studied in the recent literature. However, despite the rich results under various on-policy learning settings, there still lacks some essential understanding of the offline CMDP problems, in terms of both the algorithm design and the information theoretic sample complexity lower bound. In this paper, we focus on solving the CMDP problems where only offline data are available. By adopting the concept of the single-policy concentrability coefficient $C^*$, we establish a…
University of Michigan（密歇根大学）；University of Southern California（南加州大学）；Princeton University（普林斯顿大学）；Northwestern University（西北大学）
We consider online learning for episodic stochastically constrained Markov decision processes (CMDPs), which plays a central role in ensuring the safety of reinforcement learning. Here the loss function can vary arbitrarily across the episodes, and both the loss received and the budget consumption are revealed at the end of each episode. Previous works solve this problem under the restrictive assumption that the transition model of the Markov decision processes (MDPs) is known a priori and establish regret bounds that depend polynomially on the cardinalities of the state space $\mathcal{S}$ an…
Nagoya Institute of Technology（名古屋工业大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所革新智能综合研究中心）；National Institute for Materials Science（国立材料科学研究所）
We study primal-dual type stochastic optimization algorithms with non-uniform sampling. Our main theoretical contribution in this paper is to present a convergence analysis of Stochastic Primal Dual Coordinate (SPDC) Method with arbitrary sampling. Based on this theoretical framework, we propose Optimality Violation-based Sampling SPDC (ovsSPDC), a non-uniform sampling method based on Optimality Violation. We also propose two efficient heuristic variants of ovsSPDC called ovsSDPC+ and ovsSDPC++. Through intensive numerical experiments, we demonstrate that the proposed method and its variants a…
Achieving Instance-dependent Sample Complexity for Constrained Markov Decision Process
Department of Industrial Engineering & Decision Analytics, Hong Kong University of Science and Technology（香港科技大学工业工程与决策分析系）；Institute for Computational and Mathematical Engineering, Stanford University（斯坦福大学计算与数学工程研究所）；Department of Management Science & Engineering, Stanford University（斯坦福大学管理科学与工程系）
Near-Optimal Primal-Dual Algorithm for Learning Linear Mixture CMDPs with Adversarial Rewards
Department of Industrial and Systems Engineering, KAIST（韩国科学技术院工业与系统工程系）；Department of Mathematical Sciences, Seoul National University（首尔大学数学科学系）；Research Institute of Mathematics, Seoul National University（首尔大学数学研究所）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能跨学科项目）；Korea Institute for Advanced Study（韩国高等研究院）
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Pennsylvania State University（宾夕法尼亚州立大学）；Purdue University（普渡大学）
This paper considers stochastic linear bandits with general nonlinear constraints. The objective is to maximize the expected cumulative reward over horizon $T$ subject to a set of constraints in each round $τ\leq T$. We propose a pessimistic-optimistic algorithm for this problem, which is efficient in two aspects. First, the algorithm yields $\tilde{\cal O}\left(\left(\frac{K^{0.75}}δ+d\right)\sqrtτ\right)$ (pseudo) regret in round $τ\leq T,$ where $K$ is the number of constraints, $d$ is the dimension of the reward feature space, and $δ$ is a Slater's constant; and zero constraint violation i…
Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs
University of Tennessee, Knoxville（田纳西大学）；University of Maryland, College Park（马里兰大学）；Apple Inc.（苹果公司）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Southern California（南加州大学）
Constrained Markov decision processes (CMDPs) model scenarios of sequential decision making with multiple objectives that are increasingly important in many applications. However, the model is often unknown and must be learned online while still ensuring the constraint is met, or at least the violation is bounded with time. Some recent papers have made progress on this very challenging problem but either need unsatisfactory assumptions such as knowledge of a safe policy, or have high cumulative regret. We propose the Safe PSRL (posterior sampling-based RL) algorithm that does not need such ass…
Georgia Institute of Technology（佐治亚理工学院）；The Ohio State University（俄亥俄州立大学）；University at Buffalo, The State University of New York（纽约州立大学布法罗分校）
The problem of constrained Markov decision process (CMDP) is investigated, where an agent aims to maximize the expected accumulated discounted reward subject to multiple constraints on its utilities/costs. A new primal-dual approach is proposed with a novel integration of three ingredients: entropy regularized policy optimizer, dual variable regularizer, and Nesterov's accelerated gradient descent dual optimizer, all of which are critical to achieve a faster convergence. The finite-time error bound of the proposed approach is characterized. Despite the challenge of the nonconcave objective sub…
University of Southern California（南加利福尼亚大学）；Facebook Inc.（脸书公司）；Princeton University（普林斯顿大学）；Northwestern University（西北大学）
We study the Safe Reinforcement Learning (SRL) problem using the Constrained Markov Decision Process (CMDP) formulation in which an agent aims to maximize the expected total reward subject to a safety constraint on the expected total value of a utility function. We focus on an episodic setting with the function approximation where the Markov transition kernels have a linear structure but do not impose any additional assumptions on the sampling model. Designing SRL algorithms with provable computational and statistical efficiency is particularly challenging under this setting because of the nee…
This paper studies how to train machine-learning models that directly approximate the optimal solutions of constrained optimization problems. This is an empirical risk minimization under constraints, which is challenging as training must balance optimality and feasibility conditions. Supervised learning methods often approach this challenge by training the model on a large collection of pre-solved instances. This paper takes a different route and proposes the idea of Primal-Dual Learning (PDL), a self-supervised training method that does not require a set of pre-solved instances or an optimiza…
Policy-based Primal-Dual Methods for Concave CMDP with Variance Reduction
Department of Industrial Engineering and Operations Research, UC Berkeley（加州大学伯克利分校工业工程与运筹学系）；Department of Mechanical Engineering, UC Berkeley（加州大学伯克利分校机械工程系）
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）
In this paper we study the convex-concave saddle-point problem $\min_x \max_y f(x) + y^T \mathbf{A} x - g(y)$, where $f(x)$ and $g(y)$ are smooth and convex functions. We propose an Accelerated Primal-Dual Gradient Method (APDG) for solving this problem, achieving (i) an optimal linear convergence rate in the strongly-convex-strongly-concave regime, matching the lower complexity bound (Zhang et al., 2021), and (ii) an accelerated linear convergence rate in the case when only one of the functions $f(x)$ and $g(y)$ is strongly convex or even none of them are. Finally, we obtain a linearly conver…
Lehigh University（理海大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Distributed optimization algorithms are essential for training machine learning models on very large-scale datasets. However, they often suffer from communication bottlenecks. Confronting this issue, a communication-efficient primal-dual coordinate ascent framework (CoCoA) and its improved variant CoCoA+ have been proposed, achieving a convergence rate of $\mathcal{O}(1/t)$ for solving empirical risk minimization problems with Lipschitz continuous losses. In this paper, an accelerated variant of CoCoA+ is proposed and shown to possess a convergence rate of $\mathcal{O}(1/t^2)$ in terms of redu…
Trinity College Dublin（都柏林圣三一学院）；TU Delft（代尔夫特理工大学）
In this paper we extend the classical Follow-The-Regularized-Leader (FTRL) algorithm to encompass time-varying constraints, through adaptive penalization. We establish sufficient conditions for the proposed Penalized FTRL algorithm to achieve $O(\sqrt{t})$ regret and violation with respect to strong benchmark $\hat{X}^{max}_t$. Lacking prior knowledge of the constraints, this is probably the largest benchmark set that we can reasonably hope for. Our sufficient conditions are necessary in the sense that when they are violated there exist examples where $O(\sqrt{t})$ regret and violation is not …
In this paper, we develop a novel virtual-queue-based online algorithm for online convex optimization (OCO) problems with long-term and time-varying constraints and conduct a performance analysis with respect to the dynamic regret and constraint violations. We design a new update rule of dual variables and a new way of incorporating time-varying constraint functions into the dual variables. To the best of our knowledge, our algorithm is the first parameter-free algorithm to simultaneously achieve sublinear dynamic regret and constraint violations. Our proposed algorithm also outperforms the st…
Australian National University（澳大利亚国立大学）；Data61；University of Sydney（悉尼大学）
Since the introduction of Generative Adversarial Networks (GANs) and Variational Autoencoders (VAE), the literature on generative modelling has witnessed an overwhelming resurgence. The impressive, yet elusive empirical performance of GANs has lead to the rise of many GAN-VAE hybrids, with the hopes of GAN level performance and additional benefits of VAE, such as an encoder for feature reduction, which is not offered by GANs. Recently, the Wasserstein Autoencoder (WAE) was proposed, achieving performance similar to that of GANs, yet it is still unclear whether the two are fundamentally differe…
Avoiding Structural Failure Modes in Tabular Fair SSL: Online Primal-Dual Allocation under Confidence Gating
College of Computer Science and Technology, Jilin University, China（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education, Jilin University, China（教育部符号计算与知识工程重点实验室）
