We consider policy gradient algorithms for the indefinite least squares stationary optimal control, e.g., linear-quadratic-regulator (LQR) with indefinite state and input penalization matrices. Such a setup has important applications in control design with conflicting objectives, such as linear quadratic dynamic games. We show the global convergence of gradient, natural gradient and quasi-Newton policies for this class of indefinite least squares problems.
We propose projection-free sequential algorithms for linear-quadratic dynamics games. These policy gradient based algorithms are akin to Stackelberg leadership model and can be extended to model-free settings. We show that if the leader performs natural gradient descent/ascent, then the proposed algorithm has a global sublinear convergence to the Nash equilibrium. Moreover, if the leader adopts a quasi-Newton policy, the algorithm enjoys a global quadratic convergence. Along the way, we examine and clarify the intricacies of adopting sequential policy updates for LQ games, namely, issues perta…
This paper studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques. Based on policy iteration, a novel off-policy reinforcement learning algorithm, named optimistic least-squares-based policy iteration, is proposed which is able to find iteratively near-optimal policies of the adaptive optimal stationary control problem directly from input/state data without explicitly identifying any system matrices, starting from an initial admissible control policy. The solutions given …
INRIA Nancy Grand Est（法国国家信息与自动化研究所南锡-大东部大区中心）；CNRS（法国国家科学研究中心）；Université de Lorraine（洛林大学）
We consider approximate dynamic programming for the infinite-horizon stationary $γ$-discounted optimal control problem formalized by Markov Decision Processes. While in the exact case it is known that there always exists an optimal policy that is stationary, we show that when using value function approximation, looking for a non-stationary policy may lead to a better performance guarantee. We define a non-stationary variant of MPI that unifies a broad family of approximate DP algorithms of the literature. For this algorithm we provide an error propagation analysis in the form of a performance …
In this paper, we consider a finite horizon non-stationary inventory system with setup costs. We detail an algorithm to find an approximate optimal policy and the basic idea is to use numerical procedure for computing integrals involved in the standard method. We provide analytical error bounds, which converge to zero, between the costs of an approximate optimal policy found in this paper and the optimal policy, which are the main contributions of this paper. To the best of our knowledge, the error bound results are not found in the literature on finite horizon inventory systems with setup cos…
This paper addresses the single-item single-stocking location non-stationary stochastic lot-sizing problem under a reorder point -- order quantity control strategy. The reorder points and order quantities are chosen at the beginning of the planning horizon. The reorder points are allowed to vary with time and we consider order quantities either to be a series of time-dependent constants or a fixed value; this leads to two variants of the policy: the (st,Qt) and the (st,Q) policies, respectively. For both policies, we present stochastic dynamic programs (SDP) to determine optimal policy paramet…
Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
Department of Industrial Engineering and Operations Research, Columbia University（工业工程与运营管理系，哥伦比亚大学）；Data Science Institute, Columbia University（数据科学研究所，哥伦比亚大学）
针对连续时间随机 LQ 控制的无模型 RL，本文提出由 critic 与 actor 自适应调节熵正则化和策略方差的数据驱动探索机制，以少量调参实现匹配已知最优固定调度结果的次线性遗憾，并在实验中加速收敛、降低遗憾。
First order policy optimization has been widely used in reinforcement learning. It guarantees to find the optimal policy for the state-feedback linear quadratic regulator (LQR). However, the performance of policy optimization remains unclear for the linear quadratic Gaussian (LQG) control where the LQG cost has spurious suboptimal stationary points. In this paper, we introduce a novel perturbed policy gradient (PGD) method to escape a large class of bad stationary points (including high-order saddles). In particular, based on the specific structure of LQG, we introduce a novel reparameterizati…
We study the synthesis of a policy in a Markov decision process (MDP) following which an agent reaches a target state in the MDP while minimizing its total discounted cost. The problem combines a reachability criterion with a discounted cost criterion and naturally expresses the completion of a task with probabilistic guarantees and optimal transient performance. We first establish that an optimal policy for the considered formulation may not exist but that there always exists a near-optimal stationary policy. We additionally provide a necessary and sufficient condition for the existence of an…
We consider reinforcement learning (RL) in episodic MDPs with adversarial full-information reward feedback and unknown fixed transition kernels. We propose two model-free policy optimization algorithms, POWER and POWER++, and establish guarantees for their dynamic regret. Compared with the classical notion of static regret, dynamic regret is a stronger notion as it explicitly accounts for the non-stationarity of environments. The dynamic regret attained by the proposed algorithms interpolates between different regimes of non-stationarity, and moreover satisfies a notion of adaptive (near-)opti…
For a Markov decision process with countably infinite states, the optimal value may not be achievable in the set of stationary policies. In this paper, we study the existence conditions of an optimal stationary policy in a countable-state Markov decision process under the long-run average criterion. With a properly defined metric on the policy space of ergodic MDPs, the existence of an optimal stationary policy can be guaranteed by the compactness of the space and the continuity of the long-run average cost with respect to the metric. We further extend this condition by some assumptions which …
This paper addresses the two-stocking locations single item non-stationary stochastic lot-sizing problem. The inventory level at each location is reviewed periodically. Items can be reordered and received from a common central warehouse and can also be transshipped laterally from the other location. Lateral transshipment is assumed to be proactive to re-distribute the stock between two stocking locations. Therefore, the order of action in each period is: transshipping (if necessary), reordering (if necessary) and satisfying the demand at each location and each installation. The costs are impos…
This paper addresses the single-item single-stocking location stochastic lot sizing problem under the $(s, S) $ policy. We first present a mixed integer non-linear programming (MINLP) formulation for determining near-optimal $(s, S)$ policy parameters. To tackle larger instances, we then combine the previously introduced MINLP model and a binary search approach. These models can be reformulated as mixed integer linear programming (MILP) models which can be easily implemented and solved by using off-the-shelf optimisation software. Computational experiments demonstrate that optimality gaps of t…
We study the optimal transport problem for pairs of stationary finite-state Markov chains, with an emphasis on the computation of optimal transition couplings. Transition couplings are a constrained family of transport plans that capture the dynamics of Markov chains. Solutions of the optimal transition coupling (OTC) problem correspond to alignments of the two chains that minimize long-term average cost. We establish a connection between the OTC problem and Markov decision processes, and show that solutions of the OTC problem can be obtained via an adaptation of policy iteration. For settings…
Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at all. We answer both the questions affirmatively in the overparameterized regime. In detail, we prove that neural natural policy gradient converges to a globally optimal policy at a sublinear rate. Also, we show that neural vanilla policy gradient converges sublinearly to a statio…
We consider the determination of the optimal stationary singular stochastic control of a linear diffusion for a class of average cumulative cost minimization problems arising in various financial and economic applications of stochastic control theory. We present a set of conditions under which the optimal policy is of the standard local time reflection type and state the first order conditions from which the boundaries can be determined. Since the conditions do not require symmetry or convexity of the costs, our results cover also the cases where costs are asymmetric and non-convex. We also in…
Max Planck Institute for Mathematics in the Sciences（马克斯·普朗克数学研究所）；Faculty of Mathematics and Computer Science（数学与计算机科学学院）；Leipzig University（莱比锡大学）；Santa Fe Institute（圣塔菲研究所）
It is well known that for any finite state Markov decision process (MDP) there is a memoryless deterministic policy that maximizes the expected reward. For partially observable Markov decision processes (POMDPs), optimal memoryless policies are generally stochastic. We study the expected reward optimization problem over the set of memoryless stochastic policies. We formulate this as a constrained linear optimization problem and develop a corresponding geometric framework. We show that any POMDP has an optimal memoryless policy of limited stochasticity, which allows us to reduce the dimensional…
Shanghai University of Finance and Economics（上海财经大学）；Fudan University（复旦大学）；Stanford University（斯坦福大学）
In this paper, we propose several new stochastic second-order algorithms for policy optimization that only require gradient and Hessian-vector product in each iteration, making them computationally efficient and comparable to policy gradient methods. Specifically, we propose a dimension-reduced second-order method (DR-SOPO) which repeatedly solves a projected two-dimensional trust region subproblem. We show that DR-SOPO obtains an $\mathcal{O}(ε^{-3.5})$ complexity for reaching approximate first-order stationary condition and certain subspace second-order stationary condition. In addition, we …
This paper studies a distributed policy gradient in collaborative multi-agent reinforcement learning (MARL), where agents over a communication network aim to find the optimal policy to maximize the average of all agents' local returns. Due to the non-concave performance function of policy gradient, the existing distributed stochastic optimization methods for convex problems cannot be directly used for policy gradient in MARL. This paper proposes a distributed policy gradient with variance reduction and gradient tracking to address the high variances of policy gradient, and utilizes importance …
Deep reinforcement learning (RL) has made it possible to solve complex robotics problems using neural networks as function approximators. However, the policies trained on stationary environments suffer in terms of generalization when transferred from one environment to another. In this work, we use Robust Markov Decision Processes (RMDP) to train the drone control policy, which combines ideas from Robust Control and RL. It opts for pessimistic optimization to handle potential gaps between policy transfer from one environment to another. The trained control policy is tested on the task of quadc…
Policy Optimization and Multi-agent Reinforcement Learning for Mean-variance Team Stochastic Games
School of Mechanical and Electrical Engineering, Shenzhen Polytechnic University（机械与电气工程学院，深圳职业技术大学）；School of Business, Sun Yat-Sen University（商学院，中山大学）
The design of an efficient curing policy, able to stem an epidemic process at an affordable cost, has to account for the structure of the population contact network supporting the contagious process. Thus, we tackle the problem of allocating recovery resources among the population, at the lowest cost possible to prevent the epidemic from persisting indefinitely in the network. Specifically, we analyze a susceptible-infected-susceptible epidemic process spreading over a weighted graph, by means of a first-order mean-field approximation. First, we describe the influence of the contact network on…
This paper presents a new condition for the existence of optimal stationary policies in average-cost continuous-time Markov decision processes with unbounded cost and transition rates, arising from controlled queueing systems. This condition is closely related to the stability of queueing systems. It suggests that the proof of the stability can be exploited to verify the existence of an optimal stationary policy. This new condition is easier to verify than existing conditions. Moreover, several conditions are provided which suffice for the average-cost optimality equality to hold.
We consider the static output feedback control for Linear Quadratic Regulator problems with structured constraints under the assumption that system parameters are unknown. To solve the problem in the model free setting, we propose the policy gradient algorithm based on the gradient projection method and show its global convergence to $\varepsilon$-stationary points. In addition, we introduce a variance reduction technique and show both theoretically and numerically that it significantly reduces the variance in the gradient estimation. We also show in the numerical experiments that the model fr…
We consider the dynamic inventory problem with non-stationary demands. It has long been known that non-stationary (s, S) policies are optimal for this problem. However, finding optimal policy parameters remains a computational challenge as it requires solving a large-scale stochastic dynamic program. To address this, we devise a recursion-free approximation for the optimal cost function of the problem. This enables us to compute policy parameters heuristically, without resorting to a stochastic dynamic program. The heuristic is easy-to-understand and -use since it follows by elementary methods…
Benign Nonconvex Landscape for Policy Optimization: Infinite-Horizon Discounted MDPs with General State and Action Spaces
Naveen Jindal School of Management, University of Texas at Dallas（德克萨斯大学达拉斯分校纳文·金达尔管理学院）；H. Milton Stewart School of Industrial and Systems Engineering, Georgia Tech（佐治亚理工学院H.米尔顿·斯图尔特工业与系统工程学院）
When designing algorithms for finite-time-horizon episodic reinforcement learning problems, a common approach is to introduce a fictitious discount factor and use stationary policies for approximations. Empirically, it has been shown that the fictitious discount factor helps reduce variance, and stationary policies serve to save the per-iteration computational cost. Theoretically, however, there is no existing work on convergence analysis for algorithms with this fictitious discount recipe. This paper takes the first step towards analyzing these algorithms. It focuses on two vanilla policy gra…
Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value. In this paper, we consider the statistically efficient estimation of policy gradients from off-policy data, where the estimation is particularly non-trivial. We derive the asymptotic lower bound on the feasible mean-squared error in both Markov and non-Markov decision processes and show that existing estimators fail to achieve it in general settings. We propose a meta-algorithm that achieves the lower bound without any parametric assumptions and …
Score-Aware Policy-Gradient and Performance Guarantees using Local Lyapunov Stability
LAAS–CNRS, Université de Toulouse, CNRS（LAAS–CNRS，图卢兹大学，CNRS）；Eindhoven University of Technology（埃因霍温理工大学）；LAAS-CNRS, IRIT, and Université de Toulouse（LAAS-CNRS，IRIT，图卢兹大学）
We develop a normative framework for hierarchical model-based policy optimization based on applying second-order methods in the space of all possible state-action paths. The resulting natural path gradient performs policy updates in a manner which is sensitive to the long-range correlational structure of the induced stationary state-action densities. We demonstrate that the natural path gradient can be computed exactly given an environment dynamics model and depends on expressions akin to higher-order successor representations. In simulation, we show that the priorization of local policy updat…
