This paper studies a class of partially observed Linear Quadratic Gaussian (LQG) problems with unknown dynamics. We establish an end-to-end sample complexity bound on learning a robust LQG controller for open-loop stable plants. This is achieved using a robust synthesis procedure, where we first estimate a model from a single input-output trajectory of finite length, identify an H-infinity bound on the estimation error, and then design a robust controller using the estimated model and its quantified uncertainty. Our synthesis procedure leverages a recent control tool called Input-Output Parameterization (IOP) that enables robust controller design using convex optimization. For open-loop stable systems, we prove that the LQG performance degrades linearly with respect to the model estimation error using the proposed synthesis procedure. Despite the hidden states in the LQG problem, the achieved scaling matches previous results on learning Linear Quadratic Regulator (LQR) controllers with full state observations.
Distributed control problems under some specific information constraints can be formulated as (possibly infinite dimensional) convex optimization problems. The underlying motivation of this work is to develop an understanding of the optimal decision making architecture for such problems. In this paper, we particularly focus on the N-player triangular LQG problems and show that the optimal output feedback controllers have attractive state space realizations. The optimal controller can be synthesized using a set of stabilizing solutions to 2N linearly coupled algebraic Riccati equations, which t…
This paper investigates the linear-quadratic-Gaussian (LQG) mean-field game (MFG) for a class of stochastic delay systems. We consider a large population system in which the dynamics of each player satisfies some forward stochastic differential delay equation (SDDE). The consistency condition or Nash certainty equivalence (NCE) principle is established through an auxiliary mean-field system of anticipated forward-backward stochastic differential equation with delay (AFBSDDE). The wellposedness of such consistency condition system can be further established by some continuation method instead t…
In this work, we revisit the Linear Quadratic Gaussian (LQG) optimal control problem from a behavioral perspective. Motivated by the suitability of behavioral models for data-driven control, we begin with a reformulation of the LQG problem in the space of input-output behaviors and obtain a complete characterization of the optimal solutions. In particular, we show that the optimal LQG controller can be expressed as a static behavioral-feedback gain, thereby eliminating the need for dynamic state estimation characteristic of state space methods. The static form of the optimal LQG gain also make…
We study the problem of adaptive control in partially observable linear quadratic Gaussian control systems, where the model dynamics are unknown a priori. We propose LqgOpt, a novel reinforcement learning algorithm based on the principle of optimism in the face of uncertainty, to effectively minimize the overall control cost. We employ the predictor state evolution representation of the system dynamics and deploy a recently proposed closed-loop system identification method, estimation, and confidence bound construction. LqgOpt efficiently explores the system dynamics, estimates the model param…
Information communicated within cyber-physical systems (CPSs) is often used in determining the physical states of such systems, and malicious adversaries may intercept these communications in order to infer future states of a CPS or its components. Accordingly, there arises a need to protect the state values of a system. Recently, the notion of differential privacy has been used to protect state trajectories in dynamical systems, and it is this notion of privacy that we use here to protect the state trajectories of CPSs. We incorporate a cloud computer to coordinate the agents comprising the C…
First order policy optimization has been widely used in reinforcement learning. It guarantees to find the optimal policy for the state-feedback linear quadratic regulator (LQR). However, the performance of policy optimization remains unclear for the linear quadratic Gaussian (LQG) control where the LQG cost has spurious suboptimal stationary points. In this paper, we introduce a novel perturbed policy gradient (PGD) method to escape a large class of bad stationary points (including high-order saddles). In particular, based on the specific structure of LQG, we introduce a novel reparameterizati…
This paper revisits the classical Linear Quadratic Gaussian (LQG) control from a modern optimization perspective. We analyze two aspects of the optimization landscape of the LQG problem: 1) connectivity of the set of stabilizing controllers $\mathcal{C}_n$; and 2) structure of stationary points. It is known that similarity transformations do not change the input-output behavior of a dynamical controller or LQG cost. This inherent symmetry by similarity transformations makes the landscape of LQG very rich. We show that 1) the set of stabilizing controllers $\mathcal{C}_n$ has at most two path-c…
This paper is concerned with a backward-forward stochastic differential equation (BFSDE) system, in which a large number of negligible agents are coupled in their dynamics via state average. Here some BSDE is introduced as the dynamics of major player, while dynamics of minor players are described by SDEs. Some auxiliary mean-field SDEs (MFSDEs) and a $3\times2$ mixed forward-backward stochastic differential equation (FBSDE) system are considered and analyzed instead of involving the fixed-point analysis as in other mean-field games. We also derive the decentralized strategies which are shown …
In this paper, we define and solve the Inverse Stochastic Optimal Control (ISOC) problem of the linear-quadratic Gaussian (LQG) and the linear-quadratic sensorimotor (LQS) control model. These Stochastic Optimal Control (SOC) models are state-of-the-art approaches describing human movements. The LQG ISOC problem consists of finding the unknown weighting matrices of the quadratic cost function and the covariance matrices of the additive Gaussian noise processes based on ground truth trajectories observed from the human in practice. The LQS ISOC problem aims at additionally finding the covarianc…
In this work, we study a class of mean-field linear quadratic Gaussian (LQG) problems. Under suitable conditions, explicit solutions of the distribution-dependent optimal control problems are obtained. Riccati systems are derived by directly solving the associated master equations. Some extensions on controls with partial observations are also considered.
This paper studies a new class of dynamic optimization problems of large-population (LP) system which consists of a large number of negligible and coupled agents. The most significant feature in our setup is the dynamics of individual agents follow the forward-backward stochastic differential equations (FBSDEs) in which the forward and backward states are coupled at the terminal time. This current paper is hence different to most existing large-population literature where the individual states are typically modeled by the SDEs including the forward state only. The associated mean-field linear-…
This paper studies social optimal control of mean field LQG (linear-quadratic-Gaussian) models with uncertainty. Specially, the uncertainty is represented by a uncertain drift which is common for all agents. A robust optimization approach is applied by assuming all agents treat the uncertain drift as an adversarial player. In our model, both dynamics and costs of agents are coupled by mean field terms, and both finite- and infinite-time horizon cases are considered. By examining social functional variation and exploiting person-by-person optimality principle, we construct an auxiliary control …
The goal of this paper is to study a multi-objective linear quadratic Gaussian (LQG) control problem. In particular, we consider an optimal control problem minimizing a quadratic cost over a finite time horizon for linear stochastic systems subject to control energy constraints. To solve the problem, we suggest an efficient bisection line search algorithm which is computationally efficient compared to other approaches such as the semidefinite programming. The main idea is to use the Lagrangian function and Karush-Kuhn-Tucker (KKT) optimality conditions to solve the constrained optimization pro…
This paper studies approximate solutions to large-scale linear quadratic stochastic games with homogeneous nodal dynamics parameters and heterogeneous network couplings within the graphon mean field game framework in [2]-[4]. A graphon time-varying dynamical system model is first formulated to study the finite and then limit problems of linear quadratic Gaussian graphon mean field games (LQG-GMFG). The Nash equilibrium of the limit problem is then characterized by two coupled graphon time-varying dynamical systems. Sufficient conditions are established for the existence of a unique solution to…
We consider a joint sensor and controller design problem for linear Gaussian stochastic systems in which a weighted sum of quadratic control cost and the amount of information acquired by the sensor is minimized. This problem formulation is motivated by situations where a control law must be designed in the presence of sensing, communication, and privacy constraints. We show that the optimal joint sensor-controller design is relatively easy when the sensing policy is restricted to be linear. Namely, an explicit form of the optimal linear sensor equation, the Kalman filter, and the certainty eq…
Optimal control theory and machine learning techniques are combined to formulate and solve in closed form an optimal control formulation of online learning from supervised examples with regularization of the updates. The connections with the classical Linear Quadratic Gaussian (LQG) optimal control problem, of which the proposed learning paradigm is a non-trivial variation as it involves random matrices, are investigated. The obtained optimal solutions are compared with the Kalman-filter estimate of the parameter vector to be learned. It is shown that the proposed algorithm is less sensitive t…
We consider the problem of finding an event-based sampling scheme that optimizes the trade-off between average sampling rate and control performance in a linear-quadratic-Gaussian (LQG) control problem setting with output feedback. Our analysis is based on a recently presented sampled-data controller structure, which remains LQG-optimal for any choice of sampling scheme. We show that optimization of the sampling scheme is related to an elliptic convection-diffusion type partial differential equation over a domain with free boundary, a so called Stefan problem. A numerical method is presented t…
Planning under motion and observation uncertainties requires solution of a stochastic control problem in the space of feedback policies. In this paper, we reduce the general (n^2+n)-dimensional belief space planning problem to an (n)-dimensional problem by obtaining a Linear Quadratic Gaussian (LQG) design with the best nominal performance. Then, by taking the underlying trajectory of the LQG controller as the decision variable, we pose a coupled design of trajectory, estimator, and controller design through a Non-Linear Program (NLP) that can be solved by a general NLP solver. We prove that u…
A code for communication over the k-receiver additive white Gaussian noise broadcast channel with feedback is presented and analyzed using tools from the theory of linear quadratic Gaussian optimal control. It is shown that the performance of this code depends on the noise correlation at the receivers and it is related to the solution of a discrete algebraic Riccati equation. For the case of independent noises, the sum rate achieved by the proposed code, satisfying average power constraint P, is characterized as 1/2 log (1+P*phi), where the coefficient "phi" in the interval [1,k] quantifies th…
A decentralized control system with linear dynamics, quadratic cost, and Gaussian disturbances is considered. The system consists of a finite number of subsystems whose dynamics and per-step cost function are coupled through their mean-field (empirical average). The system has mean-field sharing information structure, i.e., each controller observes the state of its local subsystem (either perfectly or with noise) and the mean-field. It is shown that the optimal control law is unique, linear, and identical across all subsystems. Moreover, the optimal gains are computed by solving two decoupled …
Inverse Problems for Costs and Controls in LQG MFGs via Mean Field Trajectories
Center for Data Science, New York University（纽约大学数据科学中心）；NYU Shanghai（纽约大学上海分校）；Shanghai Center for Data Science（上海数据科学中心）；NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai（纽约大学上海分校纽约-华东师范大学数学科学研究所）
We consider scalar decentralized average-cost infinite-horizon LQG problems with two controllers, focusing on the fast dynamics case when the (scalar) eigenvalue of the system is large. It is shown that the best linear controllers' performance can be an arbitrary factor worse than the optimal performance. We propose a set of finite-dimensional nonlinear controllers, and prove that the proposed set contains an easy-to-find approximately optimal solution that achieves within a constant ratio of the optimal quadratic cost. The insight for nonlinear strategies comes from revealing the relationship…
In this paper, our goal is to study fundamental foundations of linear quadratic Gaussian (LQG) control problems for stochastic linear time-invariant systems via Lagrangian duality of semidefinite programming (SDP) problems. In particular, we derive an SDP formulation of the finite-horizon LQG problem, and its Lagrangian duality. Moreover, we prove that Riccati equation for LQG can be derived the KKT optimality condition of the corresponding SDP problem. Besides, the proposed primal problem efficiently decouples the system matrices and the gain matrix. This allows us to develop new convex relax…
We consider the infinite dimensional linear programming (inf-LP) approach for solving stochastic control problems. The inf-LP corresponding to problems with uncountable state and input spaces is in general computationally intractable. By focusing on linear systems with quadratic cost (LQG), we establish a connection between this approach and the well-known Riccati LMIs. In particular, we show that the semidefinite programs known for the LQG problem can be derived from the pair of primal and dual inf-LPs. Furthermore, we establish a connection between multi-objective and chance constraint crite…
This paper is concerned with linear-quadratic-Gaussian (LQG) control for a field-mediated feedback connection of a plant and a coherent (measurement-free) controller. Both the plant and the controller are multimode open quantum harmonic oscillators governed by linear quantum stochastic differential equations. The control objective is to make the closed-loop system internally stable and to minimize the infinite-horizon quadratic cost involving the plant variables and the controller output subject to quantum physical realizability (PR) constraints. This coherent quantum LQG (CQLQG) control probl…
We propose a new framework to design controllers for high-dimensional nonlinear systems. The control is designed through the iterative linear quadratic regulator (ILQR), an algorithm that computes control by iteratively applying the linear quadratic regulator on the local linearization of the system at each time step. Since ILQR is computationally expensive, we propose to first construct reduced-order models (ROMs) of the high-dimensional nonlinear system. We derive nonlinear ROMs via projection, where the basis is computed via balanced truncation (BT) and LQG balanced truncation (LQG-BT). Num…
We consider a rod that is heated/cooled and sensed at multiple point locations. To stabilize it to a constant temperature we set up a Linear Quadratic Regulator that we explicitly solve by the method of completing the square to find the optimal linear state feedback for the point actuators. But we don't assume that the whole state is mesureable so we construct an infinite dimensional Kalman filter to estimate the whole state from a finite number of noisy point measurements. These two components yield a Linear Quadratic Gaussian (LQG) Synthesis for the heat equation under point actuation and po…
We study linear quadratic Gaussian (LQG) control design for linear port-Hamiltonian systems. To this end, we exploit the freedom in choosing the weighting matrices and propose a specific choice which leads to an LQG controller which is port-Hamiltonian and, thus, in particular stable and passive. Furthermore, we construct a reduced-order controller via balancing and subsequent truncation. This approach is closely related to classical LQG balanced truncation and shares a similar a priori error bound with respect to the gap metric. By exploiting the non-uniqueness of the Hamiltonian, we are able…
This paper is concerned with the coherent quantum linear-quadratic-Gaussian control problem of minimising an infinite-horizon mean square cost for a measurement-free field-mediated interconnection of a quantum plant with a stabilising quantum controller. The plant and the controller are multimode open quantum harmonic oscillators, governed by linear quantum stochastic differential equations and coupled to each other and the external multichannel bosonic fields in the vacuum state. We discuss an interplay between the quantum physical realizability conditions and the Luenberger structure associa…
We consider a discrete-time Linear-Quadratic-Gaussian (LQG) control problem in which Massey's directed information from the observed output of the plant to the control input is minimized while required control performance is attainable. This problem arises in several different contexts, including joint encoder and controller design for data-rate minimization in networked control systems. We show that the optimal control law is a Linear-Gaussian randomized policy. We also identify the state space realization of the optimal policy, which can be synthesized by an efficient algorithm based on semi…
We study distributed output feedback control for a heterogeneous multi-agent system (MAS), consisting of N different continuous-time linear dynamical systems. For achieving output consensus, a virtual reference model is assumed to generate the desired trajectory that the MAS is required to track and synchronize. A distinct feature of our results lies in the local optimality and robustness achieved by our proposed consensus control algorithm. In addition our study is focused on the case when the available output measurements contain only relative information from the neighboring agents and refe…
This paper examines mean field linear-quadratic-Gaussian (LQG) social optimum control with volatility-uncertain common noise. The diffusion terms in the dynamics of agents contain an unknown volatility process driven by a common noise. We apply a robust optimization approach in which all agents view volatility uncertainty as an adversarial player. Based on the principle of person-by-person optimality and a two-step-duality technique for stochastic variational analysis, we construct an auxiliary optimal control problem for a representative agent. Through solving this problem combined with a con…
We consider the problem of steering, via output feedback, the state distribution of a discrete-time, linear stochastic system from an initial Gaussian distribution to a terminal Gaussian distribution with prescribed mean and maximum covariance, subject to probabilistic path constraints on the state. The filtered state is obtained via a Kalman filter, and the problem is formulated as a deterministic convex program in terms of the distribution of the filtered state. We observe that, in the presence of constraints on the state covariance, and in contrast to classical Linear Quadratic Gaussian (LQ…
In this paper, we consider a discrete time linear quadratic Gaussian (LQG) control problem in which state information of the plant is encoded in a variable-length binary codeword at every time step, and a control input is determined based on the codewords generated in the past. We derive a lower bound of the rate achievable by the class of prefix-free codes attaining the required LQG control performance. This lower bound coincides with the infimum of a certain directed information expression, and is computable by semidefinite programming (SDP). Based on a technique by Silva et al., we also pro…
This paper considers a class of mean field linear-quadratic-Gaussian (LQG) games with model uncertainty. The drift term in the dynamics of the agents contains a common unknown function. We take a robust optimization approach where a representative agent in the limiting model views the drift uncertainty as an adversarial player. By including the mean field dynamics in an augmented state space, we solve two optimal control problems sequentially, which combined with consistent mean field approximations provides a solution to the robust game. A set of decentralized control strategies is derived by…
