We address the problem of model-free distributed stabilization of heterogeneous multi-agent systems using reinforcement learning (RL). Two algorithms are developed. The first algorithm solves a centralized linear quadratic regulator (LQR) problem without knowing any initial stabilizing gain in advance. The second algorithm builds upon the results of the first algorithm, and extends it to distributed stabilization of multi-agent systems with predefined interaction graphs. Rigorous proofs are provided to show that the proposed algorithms achieve guaranteed convergence if specific conditions hold. A simulation example is presented to demonstrate the theoretical results.
Various new implicit parameterizations for stabilizing controllers that allow one to impose structural constraints on the controller have been proposed lately. They are convex but infinite-dimensional, formulated in the frequency domain with no available efficient methods for computation. In this paper, we introduce a kernel version of the Youla parameterization to characterize the set of stabilizing controllers. It features a single affine constraint, which allows us to recast the controller parameterization as a novel robust filtering problem. This makes it possible to derive the first effic…
In this paper we consider a class of linear time invariant systems with infinitely many unstable modes. By using the parameterization of all stabilizing controllers, we show that H-infinity controllers for such systems can be computed using the techniques developed earlier for infinite dimensional plants with finitely many unstable modes. We illustrate connections between the problem solved here and an indirect method for strongly stabilizing H-infinity controller design for systems with time delays.
Weighted sensitivity minimization is studied within the framework of strongly stabilizing (stable) H-infinity controller design for a class of infinite dimensional systems. This problem has been solved by Ganesh and Pearson, [11], for finite dimensional plants using Nevanlinna-Pick interpolation. We extend their technique to a class of unstable time delay systems. Moreover, we illustrate suboptimal solutions, and their robust implementation.
This paper introduces a framework for learning a minimum-norm stabilizing controller for a system with unknown dynamics using model-free policy optimization methods. The approach begins by first designing a Control Lyapunov Function (CLF) for a (possibly inaccurate) dynamics model for the system, along with a function which specifies a minimum acceptable rate of energy dissipation for the CLF at different points in the state-space. Treating the energy dissipation condition as a constraint on the desired closed-loop behavior of the real-world system, we use penalty methods to formulate an uncon…
Stabilizing Controllers for Multi-Input, Singular Control Gain Systems
Aerospace Engineering and Engineering Mechanics, The University of Texas at Austin（德克萨斯大学奥斯汀分校航空航天工程与应用力学）；Systems and Control Engineering, Indian Institute of Technology, Bombay（印度孟买理工学院系统与控制工程）
Sydney Institute for Robotics and Intelligent Systems（悉尼机器人与智能系统研究所）；Australian Centre for Field Robotics（澳大利亚野外机器人中心）；School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney（悉尼大学航空航天、机械与机电工程学院）
This paper proposes a nonlinear policy architecture for control of partially-observed linear dynamical systems providing built-in closed-loop stability guarantees. The policy is based on a nonlinear version of the Youla parameterization, and augments a known stabilizing linear controller with a nonlinear operator from a recently developed class of dynamic neural network models called the recurrent equilibrium network (REN). We prove that RENs are universal approximators of contracting and Lipschitz nonlinear systems, and subsequently show that the the proposed Youla-REN architecture is a unive…
The design of an observer-based state feedback (OBSF) controller with guaranteed passivity properties for port-Hamiltonian systems (PHS) is addressed using linear matrix inequalities (LMIs). The observer gain is freely chosen and the LMIs conditions such that the state feedback is equivalent to control by interconnection with an input strictly passive (ISP) and/or an output strictly passive (OSP) and zero state detectable (ZSD) port-Hamiltonian controller are established. It is shown that the proposed controller exponentially stabilizes a class of infinite-dimensional PHS and asymptotically st…
An eigenvalue based framework is developed for the stability analysis and stabilization of coupled systems with time-delays, which are naturally described by delay differential algebraic equations. The spectral properties of these equations are analyzed and their stability properties are studied, taking into account the effect of small delay perturbations. Subsequently, numerical methods for stability assessment and for designing stabilizing controllers with a prescribed structure or order, based on a direct optimization approach, are briefly addressed. The effectiveness of the approach is ill…
The principal task to control dynamical systems is to ensure their stability. When the system is unknown, robust approaches are promising since they aim to stabilize a large set of plausible systems simultaneously. We study linear controllers under quadratic costs model also known as linear quadratic regulators (LQR). We present two different semi-definite programs (SDP) which results in a controller that stabilizes all systems within an ellipsoid uncertainty set. We further show that the feasibility conditions of the proposed SDPs are \emph{equivalent}. Using the derived robust controller syn…
Stability is one of the most fundamental requirements for systems synthesis. In this paper, we address the stabilization problem for unknown linear systems via policy gradient (PG) methods. We leverage a key feature of PG for Linear Quadratic Regulator (LQR), i.e., it drives the policy away from the boundary of the unstabilizing region along the descent direction, provided with an initial policy with finite cost. To this end, we discount the LQR cost with a factor, by adaptively increasing which gradient leads the policy to the stabilizing set while maintaining a finite cost. Based on the Lyap…
Declines in cost and concerns about the environmental impact of traditional generation have boosted the penetration of renewables and non-conventional distributed energy resources into the power grid. The intermittent availability of these resources causes the inertia of the power system to vary over time. As a result, there is a need to go beyond traditional controllers designed to regulate frequency under the assumption of invariant dynamics. This paper presents a learning-based framework for the design of stable controllers based on imitating datasets obtained from linear-quadratic regulato…
Combining a backstepping controller with a local stabilizer
Escola Politénica da Universidade de São Paulo（圣保罗大学政治工程学院）；Politecnico di Torino（都灵理工大学）；Université de Lyon（里昂大学）；Université Lyon 1（里昂第一大学）；CNRS, UMR 5007, LAGEP（法国国家科学研究中心，联合研究单位5007，实验室）；Gipsa-lab（吉普萨实验室）
A state space based design method is given to find strongly stabilizing controllers for multiple-input-multiple-output plants (MIMO). A sufficient condition is derived for the existence of suboptimal stable H-infinity controller in terms of linear matrix inequalities (LMIs) and the controller order is twice that of the plant. A new parameterization of strongly stabilizing controllers is determined using linear fractional transformations (LFTs).
This paper presents a method to design a min-norm Control Lyapunov Function (CLF)-based stabilizing controller for a control-affine system with uncertain dynamics using Gaussian Process (GP) regression. In order to estimate both state and input-dependent model uncertainty, we propose a novel compound kernel that captures the control-affine nature of the problem. Furthermore, by the use of GP Upper Confidence Bound analysis, we provide probabilistic bounds of the regression error, leading to the formulation of a CLF-based stability chance constraint which can be incorporated in a min-norm optim…
A controller synthesis method for state- and input-constrained nonlinear systems is presented that seeks continuous piecewise affine (CPA) Lyapunov-like functions and controllers simultaneously. Non-convex optimization problems are formulated on triangulated subsets of the admissible states that can be refined to meet primary control objectives, such as stability and safety, alongside secondary performance objectives. A multi-stage design is also given that enlarges the region of attraction (ROA) sequentially while allowing exclusive performance for each stage. A clear boundary for an invarian…
It is known that the set of internally stabilizing controller $\mathcal{C}_{\text{stab}}$ is non-convex, but it admits convex characterizations using certain closed-loop maps: a classical result is the Youla parameterization, and two recent notions are the system-level parameterization (SLP) and the input-output parameterization (IOP). In this paper, we address the existence of new convex parameterizations and discuss potential tradeoffs of each parametrization in different scenarios. Our main contributions are: 1) We reveal that only four groups of stable closed-loop transfer matrices are equ…
Self-triggered control is an improvement on event-triggered control methods. Unlike the latter, self-triggered control does not require monitoring the behavior of the system constantly. Instead, self-triggered algorithms predict the events at which the control law has to be updated before they happen, relying on system model and past information.\\ In this work, we present a self-triggered version of an event-triggered control method in which events are generated when a pseudo-Lyapunov function (PLF) associated with the system increases up to a certain limit. This approach has been shown to co…
This paper considers online switching control with a finite candidate controller pool, an unknown dynamical system, and unknown cost functions. The candidate controllers can be unstabilizing policies. We only require at least one candidate controller to satisfy certain stability properties, but we do not know which one is stabilizing. We design an online algorithm that guarantees finite-gain stability throughout the duration of its execution. We also provide a sublinear policy regret guarantee compared with the optimal stabilizing candidate controller. Lastly, we numerically test our algorithm…
Learning Provably Stabilizing Neural Controllers for Discrete-Time Stochastic Systems
Sharif University of Technology（谢里夫理工大学）；Institute of Science and Technology Austria (ISTA)（奥地利科学技术研究所（ISTA））；Massachusetts Institute of Technology（麻省理工学院）
Switching control strategies that unite a potentially high-performance but uncertified controller and a stabilizing albeit conservative controller are shown to be able to balance safety with efficiency, but have been less studied under partial observation of state. To address this gap, we propose a switching control strategy for partially observed linear-Gaussian systems with provable performance guarantees. We show that the proposed switching strategy is both safe and efficient, in the sense that: (1) the linear-quadratic cost of the system is always bounded even if the original uncertified c…
Integral reinforcement learning (IRL) was proposed in literature to obviate the requirement of drift dynamics in adaptive dynamic programming framework. Most of the online IRL schemes in literature require two sets of neural network (NNs), known as actor-critic NN and an initial stabilizing controller. Recently, for RL-based robust tracking this requirement of initial stabilizing controller and dual-approximator structure could be obviated by using a modified gradient descent-based update law containing a stabilizing term with critic-only structure. To the best of the authors' knowledge, there…
University of California at Los Angeles（加利福尼亚大学洛杉矶分校）
In this paper, we revisit the problem of learning a stabilizing controller from a finite number of demonstrations by an expert. By first focusing on feedback linearizable systems, we show how to combine expert demonstrations into a stabilizing controller, provided that demonstrations are sufficiently long and there are at least $n+1$ of them, where $n$ is the number of states of the system being controlled. When we have more than $n+1$ demonstrations, we discuss how to optimally choose the best $n+1$ demonstrations to construct the stabilizing controller. We then extend these results to a clas…
Recent advances in the reinforcement learning (RL) literature have enabled roboticists to automatically train complex policies in simulated environments. However, due to the poor sample complexity of these methods, solving RL problems using real-world data remains a challenging problem. This paper introduces a novel cost-shaping method which aims to reduce the number of samples needed to learn a stabilizing controller. The method adds a term involving a Control Lyapunov Function (CLF) -- an `energy-like' function from the model-based control literature -- to typical cost formulations. Theoreti…
ISIM1s consists of a few MATLAB functions and a script that can be used to derive stabilizing impulsive controllers for delay differential equations. This document serves as both a manual and tutorial on the functionality of the ISIM1s package. Brief background on the theoretically guaranteed stabilization scenario are provided before the primary MATLAB script is explained. The tutorial demonstrates how the package can be used to derive stabilizing impulsive controllers for delay differential equations of various complexity scales. Emphasis is placed on the role of various tuning parameters.
In this paper, we introduce the notion of periodic safety, which requires that the system trajectories periodically visit a subset of a forward-invariant safe set, and utilize it in a multi-rate framework where a high-level planner generates a reference trajectory that is tracked by a low-level controller under input constraints. We introduce the notion of fixed-time barrier functions which is leveraged by the proposed low-level controller in a quadratic programming framework. Then, we design a model predictive control policy for high-level planning with a bound on the rate of change for the r…
This paper addresses the problem of designing an optimal output feedback controller with a specified controller structure for linear time-invariant (LTI) systems to maximize the passivity level for the closed-loop system, in both continuous-time (CT) and discrete-time (DT). Specifically, the set of controllers under consideration is linearly parameterized with constrained parameters. Both input feedforward passivity (IFP) and output feedback passivity (OFP) indices are used to capture the level of passivity. Given a set of stabilizing controllers, a necessary and sufficient condition is propos…
This paper generalises results of Willems-Trentelman, and van der Schaft, on achievable behaviours, to the case of linear distributed systems defined by partial differential or difference equations. It shows that the `minimal' controller which achieves a particular subsystem is the canonical controller of van der Schaft, thereby answering the `open problem' of \cite{sc} in the setting of infinite dimensional and $n-D$ systems. This result is used to describe the collection of all linear subsystems of the electro-magnetic field, containing the vacuum solutions, that can be achieved by suitable …
