This paper studies the robustness of policy iteration in the context of continuous-time infinite-horizon linear quadratic regulation (LQR) problem. It is shown that Kleinman's policy iteration algorithm is inherently robust to small disturbances and enjoys local input-to-state stability in the sense of Sontag. More precisely, whenever the disturbance-induced input term in each iteration is bounded and small, the solutions of the policy iteration algorithm are also bounded and enter a small neighborhood of the optimal solution of the LQR problem. Based on this result, an off-policy data-driven policy iteration algorithm for the LQR problem is shown to be robust when the system dynamics are subjected to small additive unknown bounded disturbances. The theoretical results are validated by a numerical example.
This paper studies the finite-horizon linear quadratic regulation problem where the dynamics of the system are assumed to be unknown and the state is accessible. Information on the system is given by a finite set of input-state data, where the input injected in the system is persistently exciting of a sufficiently high order. Using data, the optimal control law is then obtained as the solution of a suitable semidefinite program. The effectiveness of the approach is illustrated via numerical examples.
This paper studies the robustness of reinforcement learning algorithms to errors in the learning process. Specifically, we revisit the benchmark problem of discrete-time linear quadratic regulation (LQR) and study the long-standing open question: Under what conditions is the policy iteration method robustly stable from a dynamical systems perspective? Using advanced stability results in control theory, it is shown that policy iteration for LQR is inherently robust to small errors in the learning process and enjoys small-disturbance input-to-state stability: whenever the error in each iteration…
University of Texas at Dallas（得克萨斯大学达拉斯分校）；University of Melbourne（墨尔本大学）
We present a midpoint policy iteration algorithm to solve linear quadratic optimal control problems in both model-based and model-free settings. The algorithm is a variation of Newton's method, and we show that in the model-based setting it achieves cubic convergence, which is superior to standard policy iteration and policy gradient algorithms that achieve quadratic and linear convergence, respectively. We also demonstrate that the algorithm can be approximately implemented without knowledge of the dynamics model by using least-squares estimates of the state-action value function from traject…
Data-Enabled Policy and Value Iteration for Continuous-Time Linear Quadratic Output Feedback Control
College of Artificial Intelligence, Nankai University（南开大学人工智能学院）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能重点实验室）
Optimal Output Feedback Learning Control for Discrete-Time Linear Quadratic Regulation
School of Automation, Beijing Institute of Technology, Beijing 100081, China（自动化学院，北京理工大学，北京）；Department of Chemical Engineering, Queen’s University, Kingston, ON K7L 3N6, Canada（化学工程系，皇后大学，加拿大）
Linear quadratic regulation of polytopic time-inhomogeneous Markov jump linear systems (extended version)
IMT School for Advanced Studies Lucca（卢卡高等研究学院）；Center of Excellence DEWS and with the Department of Information Engineering, Computer Science and Mathematics of the University of L’Aquila（拉奎拉大学信息与计算机工程及数学卓越中心）；Department of Computer Science of the University of Oxford（牛津大学计算机科学系）
Shared Linear Quadratic Regulation Control: A Reinforcement Learning Approach
MIT Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）；Laboratory of Information and Decision Systems, Massachusetts Institute of Technology（麻省理工学院信息与决策系统实验室）
We propose a methodology for performing risk-averse quadratic regulation of partially observed Linear Time-Invariant (LTI) systems disturbed by process and output noise. To compensate against the induced variability due to both types of noises, state regulation is subject to two risk constraints. The latter renders the resulting controller cautious of stochastic disturbances, by restricting the statistical variability, namely, a simplified version of the cumulative expected predictive variance of both the state and the output. Our proposed formulation results in an optimal risk-averse policy t…
We consider the continuous-time Linear-Quadratic-Regulator (LQR) problem in terms of optimizing a real-valued matrix function over the set of feedback gains. The results developed are in parallel to those in Bu et al. [1] for discrete-time LTI systems. In this direction, we characterize several analytical properties (smoothness, coerciveness, quadratic growth) that are crucial in the analysis of gradient-based algorithms. We also point out similarities and distinctive features of the continuous time setup in comparison with its discrete time analogue. First, we examine three types of well-pose…
We propose a policy iteration algorithm for solving the multiplicative noise linear quadratic output feedback design problem. The algorithm solves a set of coupled Riccati equations for estimation and control arising from a partially observable Markov decision process (POMDP) under a class of linear dynamic control policies. We show in numerical experiments far faster convergence than a value iteration algorithm, formerly the only known algorithm for solving this class of problem. The results suggest promising future research directions for policy optimization algorithms in more general POMDPs…
Recent advances in data sensing and processing technologies enable data-driven control of high-voltage direct-current (HVDC) systems for improving the operational stability of interfacing power grids. This paper proposes an optimal data-driven control strategy for an HVDC system with line-commutated converters (LCCs), wherein the dc-link voltage and current are optimally regulated at distinct HVDC terminals to improve frequency regulation (FR) in both rectifier- and inverter-side grids. Each HVDC converter is integrated with feedback loops for regulation of grid frequency and dc-link voltage i…
Declines in cost and concerns about the environmental impact of traditional generation have boosted the penetration of renewables and non-conventional distributed energy resources into the power grid. The intermittent availability of these resources causes the inertia of the power system to vary over time. As a result, there is a need to go beyond traditional controllers designed to regulate frequency under the assumption of invariant dynamics. This paper presents a learning-based framework for the design of stable controllers based on imitating datasets obtained from linear-quadratic regulato…
High-voltage direct-current (HVDC) systems for constant or intermittent power delivery have recently been developed further to support grid frequency regulation (GFR). This paper proposes a new control strategy for a line-commutated converter-based (LCC) HVDC system, wherein the DC-link voltage and current are optimally regulated to improve real-time GFR in both rectifier- and inverter-side AC networks. A dynamic model of an LCC HVDC system is developed using the DC voltage and current as input variables, and is integrated with feedback loops for inertia emulation and droop control. A linear q…
