Linear Quadratic Regulator (LQR) design is one of the most classical optimal control problems, whose well-known solution is an input sequence expressed as a state-feedback. In this work, finite-horizon and discrete-time LQR is solved under stability constraints and uncertain system dynamics. The resulting feedback controller balances cost value and closed-loop stability. Robustness of the solution is modeled using the scenario approach, without requiring any probabilistic description of the uncertainty in the system matrices. The new methods are tested and compared on the Leslie growth model, where we control population size while minimizing a suitable finite-horizon cost function.
In this paper, a polynomial chaos based framework for designing controllers for discrete time linear systems with probabilistic parameters is presented. Conditions for exponential-mean-square stability for such systems are derived and algorithms for synthesizing optimal quadratically stabilizing controllers are proposed in a convex optimization formulation. The solution presented is demonstrated on the derived discrete-time models of a nonlinear F-16 aircraft model trimmed at a set of chosen points.
The linear quadratic regulator (LQR) problem is a cornerstone of automatic control, and it has been widely studied in the data-driven setting. The various data-driven approaches can be classified as indirect (i.e., based on an identified model) versus direct or as robust (i.e., taking uncertainty into account) versus certainty-equivalence. Here we show how to bridge these different formulations and propose a novel, direct, and regularized formulation. We start from indirect certainty-equivalence LQR, i.e., least-square identification of state-space matrices followed by a nominal model-based de…
The linear-quadratic regulator (LQR) is an efficient control method for linear and linearized systems. Typically, LQR is implemented in minimal coordinates (also called generalized or "joint" coordinates). However, other coordinates are possible and recent research suggests that there may be numerical and control-theoretic advantages when using higher-dimensional non-minimal state parameterizations for dynamical systems. One such parameterization is maximal coordinates, in which each link in a multi-body system is parameterized by its full six degrees of freedom and joints between links are mo…
This paper studies the robustness of reinforcement learning algorithms to errors in the learning process. Specifically, we revisit the benchmark problem of discrete-time linear quadratic regulation (LQR) and study the long-standing open question: Under what conditions is the policy iteration method robustly stable from a dynamical systems perspective? Using advanced stability results in control theory, it is shown that policy iteration for LQR is inherently robust to small errors in the learning process and enjoys small-disturbance input-to-state stability: whenever the error in each iteration…
This paper studies the robustness of policy iteration in the context of continuous-time infinite-horizon linear quadratic regulation (LQR) problem. It is shown that Kleinman's policy iteration algorithm is inherently robust to small disturbances and enjoys local input-to-state stability in the sense of Sontag. More precisely, whenever the disturbance-induced input term in each iteration is bounded and small, the solutions of the policy iteration algorithm are also bounded and enter a small neighborhood of the optimal solution of the LQR problem. Based on this result, an off-policy data-driven …
North Carolina State University（北卡罗来纳州立大学）；Oklahoma State University（俄克拉荷马州立大学）；US Army Research Laboratory（美国陆军研究实验室）
Individual agents in a multi-agent system (MAS) may have decoupled open-loop dynamics, but a cooperative control objective usually results in coupled closed-loop dynamics thereby making the control design computationally expensive. The computation time becomes even higher when a learning strategy such as reinforcement learning (RL) needs to be applied to deal with the situation when the agents dynamics are not known. To resolve this problem, we propose a parallel RL scheme for a linear quadratic regulator (LQR) design in a continuous-time linear MAS. The idea is to exploit the structural prope…
In this paper we propose a new computational method for designing optimal regulators for high-dimensional nonlinear systems. The proposed approach leverages physics-informed machine learning to solve high-dimensional Hamilton-Jacobi-Bellman equations arising in optimal feedback control. Concretely, we augment linear quadratic regulators with neural networks to handle nonlinearities. We train the augmented models on data generated without discretizing the state space, enabling application to high-dimensional problems. We use the proposed method to design a candidate optimal regulator for an uns…
We consider adaptive control of the Linear Quadratic Regulator (LQR), where an unknown linear system is controlled subject to quadratic costs. Leveraging recent developments in the estimation of linear systems and in robust controller synthesis, we present the first provably polynomial time algorithm that provides high probability guarantees of sub-linear regret on this problem. We further study the interplay between regret minimization and parameter estimation by proving a lower bound on the expected regret in terms of the exploration schedule used by any algorithm. Finally, we conduct a nume…
We consider the classic stochastic linear quadratic regulator (LQR) problem under an infinite horizon average stage cost. By leveraging recent policy gradient methods from reinforcement learning, we obtain a first-order method that finds a stable feedback law whose objective function gap to the optima is at most $\varepsilon$ with high probability using $\tilde{O}(1/\varepsilon)$ samples, where $\tilde{O}$ hides polylogarithmic dependence on $\varepsilon$. Our proposed method seems to have the best dependence on $\varepsilon$ within the model-free literature without the assumption that all pol…
We propose a new risk-constrained reformulation of the standard Linear Quadratic Regulator (LQR) problem. Our framework is motivated by the fact that the classical (risk-neutral) LQR controller, although optimal in expectation, might be ineffective under relatively infrequent, yet statistically significant (risky) events. To effectively trade between average and extreme event performance, we introduce a new risk constraint, which explicitly restricts the total expected predictive variance of the state penalty by a user-prescribed level. We show that, under rather minimal conditions on the proc…
This paper presents a simulation-based comparison between the two controllers, Proportional Integral Derivative (PID), a classical controller and Linear Quadratic Regulator (LQR), an optimal controller, for a linearized quadrotor model. To simplify an otherwise complicated dynamic model of a quadrotor, we derive a linear mathematical model using Newtonian and Euler's laws and applying basic principles of physics. This derivation gives the equations that govern the motion of a quadrotor, both concerning the body frame and the inertial frame. A state-space model is developed, which is then used …
We present D-Phi iteration: an algorithm for distributed, localized, and scalable robust control of systems with structured uncertainties. This algorithm combines the System Level Synthesis (SLS) parametrization for distributed control with stability criteria from L1, LInf, and nu robust control. We show in simulation that this algorithm achieves good nominal performance while greatly increasing the robust stability margin compared to the LQR controller. To the best of our knowledge, this is the first distributed and localized algorithm for structured robust control; furthermore, algorithm com…
This paper studies a class of partially observed Linear Quadratic Gaussian (LQG) problems with unknown dynamics. We establish an end-to-end sample complexity bound on learning a robust LQG controller for open-loop stable plants. This is achieved using a robust synthesis procedure, where we first estimate a model from a single input-output trajectory of finite length, identify an H-infinity bound on the estimation error, and then design a robust controller using the estimated model and its quantified uncertainty. Our synthesis procedure leverages a recent control tool called Input-Output Parame…
We consider the continuous-time Linear-Quadratic-Regulator (LQR) problem in terms of optimizing a real-valued matrix function over the set of feedback gains. The results developed are in parallel to those in Bu et al. [1] for discrete-time LTI systems. In this direction, we characterize several analytical properties (smoothness, coerciveness, quadratic growth) that are crucial in the analysis of gradient-based algorithms. We also point out similarities and distinctive features of the continuous time setup in comparison with its discrete time analogue. First, we examine three types of well-pose…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Policy optimization (PO) is a key ingredient for reinforcement learning (RL). For control design, certain constraints are usually enforced on the policies to optimize, accounting for either the stability, robustness, or safety concerns on the system. Hence, PO is by nature a constrained (nonconvex) optimization in most cases, whose global convergence is challenging to analyze in general. More importantly, some constraints that are safety-critical, e.g., the $\mathcal{H}_\infty$-norm constraint that guarantees the system robustness, are difficult to enforce as the PO methods proceed. Recently, …
We introduce a robust optimization model consisting in a family of perturbation functions giving rise to certain pairs of dual optimization problems in which the dual variable depends on the uncertainty parameter. The interest of our approach is illustrated by some examples, including uncertain conic optimization and infinite optimization via discretization. The main results characterize desirable robust duality relations (as robust zero-duality gap) by formulas involving the epsilon-minima or the epsilon-subdifferentials of the objective function. The two extreme cases, namely, the usual pert…
Despite decades of research and recent progress in adaptive control and reinforcement learning, there remains a fundamental lack of understanding in designing controllers that provide robustness to inherent non-asymptotic uncertainties arising from models estimated with finite, noisy data. We propose a robust adaptive control algorithm that explicitly incorporates such non-asymptotic uncertainties into the control design. The algorithm has three components: (1) a least-squares nominal model estimator; (2) a bootstrap resampling method that quantifies non-asymptotic variance of the nominal mode…
In this work, we propose the design and analysis of a novel continuous robust controller for a class of multi--input multi--output (MIMO) nonlinear uncertain systems. The systems under consideration contains unstructured uncertainties in their drift and input matrices. The proposed controller compensates the overall system uncertainties and achieves asymptotic tracking where only the sign of the leading minors of the input gain matrix is assumed to be known. A Lyapunov based argument backed up with an integral inequality is applied to prove stability of the proposed controller and asymptotic c…
California Institute of Technology（加州理工学院）；Massachusetts Institute of Technology（麻省理工学院）
We present Neural Stochastic Contraction Metrics (NSCM), a new design framework for provably-stable robust control and estimation for a class of stochastic nonlinear systems. It uses a spectrally-normalized deep neural network to construct a contraction metric, sampled via simplified convex optimization in the stochastic setting. Spectral normalization constrains the state-derivatives of the metric to be Lipschitz continuous, thereby ensuring exponential boundedness of the mean squared distance of system trajectories under stochastic disturbances. The NSCM framework allows autonomous agents to…
This work discusses a novel framework for simultaneous synthesis of optimal watermarking signal and robust controllers in cyber-physical systems to minimize the loss in performance due to added watermarking signal and to maximize the detection rate of the attack. A general dynamic controller is designed to improve system performance with respect to the $\mathcal H_2$ norm, while a watermarking signal is added to improve security performance concerning the detection rate of replay attacks. The attack model considered in the paper is a replay attack, a natural attack mode when the dynamics of th…
This paper addresses modeling and control of a six-degree-of-freedom unmanned aerial vehicle capable of vertical take-off and landing in the presence of wind disturbances. We design a hybrid vehicle that combines the benefits of both the fixed-wing and the rotary-wing UAVs. A non-linear model for the hybrid vehicle is rapidly built, combining rigid body dynamics, aerodynamics of wing, and dynamics of the motor and propeller. Further, we design an H2 optimal controller to make the UAV robust to wind disturbances. We compare its results against that of PID and LQR-based control. Our proposed con…
An approach to stabilization of control systems with ultimately wide ranges of uncertainly disturbed parameters is offered. The method relies on using of nonlinear structurally stable functions from catastrophe theory as controllers. Analytical part presents an analysis of designed nonlinear second-order control systems. As more important the integrators in series, canonical controllable form and Jordan forms are considered. The analysis resumes that due to added controllers systems become stable and insensitive to any disturbance of parameters. Experimental part presents MATLAB simulation of …
A well known result states that stability criterion for matchings in two-sided markets doesn't ensure uniqueness. This opens the door for a moral question with regard to the optimal stable matching from a social point of view. Here, a new notion of social optimality is proposed. Its novelty is its ability to take into account the possibility of agents to leave the matching after it has already been established. To formalize this real-life scenario, this work includes a well-defined probability model and a social cost function that maintain the general guidelines of leaving-agents situations. F…
National University of Defense Technology（国防科技大学）；Jilin University（吉林大学）
Robust model predictive control (MPC) is a well-known control technique for model-based control with constraints and uncertainties. In classic robust tube-based MPC approaches, an open-loop control sequence is computed via periodically solving an online nominal MPC problem, which requires prior model information and frequent access to onboard computational resources. In this paper, we propose an efficient robust MPC solution based on receding horizon reinforcement learning, called r-LPC, for unknown nonlinear systems with state constraints and disturbances. The proposed r-LPC utilizes a Koopma…
