This paper studies a large number of homogeneous Markov decision processes where the transition probabilities and costs are coupled in the empirical distribution of states (also called mean-field). The state of each process is not known to others, which means that the information structure is fully decentralized. The objective is to minimize the average cost, defined as the empirical mean of individual costs, for which a sub-optimal solution is proposed. This solution does not depend on the number of processes, yet it converges to the optimal solution of the so-called mean-field sharing as the number of processes tends to infinity. Under some mild conditions, it is shown that the convergence rate of the proposed decentralized solution is proportional to the square root of the inverse of the number of processes. Finding this sub-optimal solution involves a non-smooth non-convex optimization problem over an uncountable set, in general. To overcome this drawback, a combinatorial optimization problem is introduced that achieves the same rate of convergence.
In this paper we study the optimality of the certainty equivalence approximation in robust finite-horizon optimization problems with expected cost. We provide an algorithm for determining the subset of the state-space for which the certainty equivalence technique is optimal. In the second part of the paper we show how patterns in the problem structure called symmetries can be used to reduce the computational complexity of the previous algorithm. Finally we demonstrate our technique through numerical examples. In particular we examine networked battery systems and radiant slab building control,…
Mean field games with a major player were introduced in (Huang, 2010) within a linear-quadratic (LQ) modeling framework. Due to the rich structure of major-minor player models, the past ten years have seen significant research efforts for different solution notions and analytical techniques. For LQ models, we address the relation between three solution frameworks: the Nash certainty equivalence (NCE) approach in (Huang, 2010), master equations, and asymptotic solvability, which have been developed starting with different ideas. We establish their equivalence relationships.
The linear quadratic regulator (LQR) problem is a cornerstone of automatic control, and it has been widely studied in the data-driven setting. The various data-driven approaches can be classified as indirect (i.e., based on an identified model) versus direct or as robust (i.e., taking uncertainty into account) versus certainty-equivalence. Here we show how to bridge these different formulations and propose a novel, direct, and regularized formulation. We start from indirect certainty-equivalence LQR, i.e., least-square identification of state-space matrices followed by a nominal model-based de…
The cooperative output regulation problem of linear multi-agent systems was formulated and studied by the distributed observer approach in [20, 21]. Since then, several variants and extensions have been proposed, and the technique of the distributed observer has also been applied to such problems as formation, rendezvous, flocking, etc. In this chapter, we will first present a more general formulation of the cooperative output regulation problem for linear multi-agent systems that includes some existing versions of the cooperative output regulation problem as special cases. Then, we will descr…
Mean field game (MFG) systems consisting of a major agent and a large number of minor agents were introduced in (Huang, 2010) in an LQG setup. The Nash certainty equivalence was used to obtain a Markovian closed-loop Nash equilibrium for the limiting system when the number of minor agents tends to infinity. In the past years several approaches to major--minor mean field game problems have been developed, principally (i) the Nash certainty equivalence and analytic approach, (ii) master equations, (iii) asymptotic solvability, and (iv) the probabilistic approach. For the LQG case, the recent wor…
This work theoretically studies a ubiquitous reinforcement learning policy for controlling the canonical model of continuous-time stochastic linear-quadratic systems. We show that randomized certainty equivalent policy addresses the exploration-exploitation dilemma in linear control systems that evolve according to unknown stochastic differential equations and their operating cost is quadratic. More precisely, we establish square-root of time regret bounds, indicating that randomized certainty equivalent policy learns optimal control actions fast from a single state trajectory. Further, linear…
The optimal adaptive control of a linear system in a signal-plus-noise setting with infinite horizon LQ regulator cost is studied. The class of partially observed linear systems for which the certainty equivalence property holds is identified. It is also shown that a linear system is adaptively stabilizable if and only if it is uniformly stabilizable, and the class of partially observed linear systems for which the certainty equivalence value function is a supersolution of the Bellman equation is identified.
We study the performance of the certainty equivalent controller on Linear Quadratic (LQ) control problems with unknown transition dynamics. We show that for both the fully and partially observed settings, the sub-optimality gap between the cost incurred by playing the certainty equivalent controller on the true system and the cost incurred by using the optimal LQ controller enjoys a fast statistical rate, scaling as the square of the parameter error. To the best of our knowledge, our result is the first sub-optimality guarantee in the partially observed Linear Quadratic Gaussian (LQG) setting.…
Massachusetts Institute of Technology（麻省理工学院）；Uppsala University（乌普萨拉大学）
Recent work by Mania et al. has proved that certainty equivalent control achieves nearly optimal regret for linear systems with quadratic costs. However, when parameter uncertainty is large, certainty equivalence cannot be relied upon to stabilize the true, unknown system. In this paper, we present a dual control strategy that attempts to combine the performance of certainty equivalence, with the practical utility of robustness. The formulation preserves structure in the representation of parametric uncertainty, which allows the controller to target reduction of uncertainty in the parameters t…
Optimality of Symmetric Independent Policies under Decentralized Mean-Field Information Sharing for Stochastic Teams and Equivalence with McKean-Vlasov Control of a Representative Agent
Royal Military College of Canada（加拿大皇家军事学院）；Bilkent University（比尔肯特大学）；Queen’s University（女王大学）
In this article, we introduce decentralized Kalman filters for linear quadratic deep structured teams. The agents in deep structured teams are coupled in dynamics, costs and measurements through a set of linear regressions of the states and actions (also called deep states and deep actions). The information structure is decentralized, where every agent observes a noisy measurement of its local state and the global deep state. Since the number of agents is often very large in deep structured teams, any naive approach to finding an optimal Kalman filter suffers from the curse of dimensionality. …
We consider a joint sensor and controller design problem for linear Gaussian stochastic systems in which a weighted sum of quadratic control cost and the amount of information acquired by the sensor is minimized. This problem formulation is motivated by situations where a control law must be designed in the presence of sensing, communication, and privacy constraints. We show that the optimal joint sensor-controller design is relatively easy when the sensing policy is restricted to be linear. Namely, an explicit form of the optimal linear sensor equation, the Kalman filter, and the certainty eq…
This paper investigates the linear-quadratic-Gaussian (LQG) mean-field game (MFG) for a class of stochastic delay systems. We consider a large population system in which the dynamics of each player satisfies some forward stochastic differential delay equation (SDDE). The consistency condition or Nash certainty equivalence (NCE) principle is established through an auxiliary mean-field system of anticipated forward-backward stochastic differential equation with delay (AFBSDDE). The wellposedness of such consistency condition system can be further established by some continuation method instead t…
Mean-Field Systems with Heterogeneous Subteams: Optimality of Cluster-Symmetric Independent Policies and Equivalence with Decentralized McKean-Vlasov Control of Cluster-Representative Agents
Isomorphism Properties of Optimality and Equilibrium Solutions under Equivalent Information Structure Transformations: Stochastic Dynamic Games and Teams
Texas A & M University（德克萨斯农工大学）；Indian Institute of Science（印度科学学院）
We consider the problem of controlling an unknown stochastic linear system with quadratic costs - called the adaptive LQ control problem. We re-examine an approach called ''Reward Biased Maximum Likelihood Estimate'' (RBMLE) that was proposed more than forty years ago, and which predates the ''Upper Confidence Bound'' (UCB) method as well as the definition of ''regret'' for bandit problems. It simply added a term favoring parameters with larger rewards to the criterion for parameter estimation. We show how the RBMLE and UCB methods can be reconciled, and thereby propose an Augmented RBMLE-UCB …
Separated design of encoder and controller for networked linear quadratic optimal control
School of Computing and Electrical Engineering, Indian Institute of Technology, Mandi, India（印度曼迪印度理工学院计算与电气工程学院）；School of Electrical Engineering, Royal Institute of Technology, Stockholm, Sweden（瑞典斯德哥尔摩皇家理工学院电气工程学院）
We propose a certainty-equivalence scheme for adaptive control of scalar linear systems subject to additive, i.i.d. Gaussian disturbances and bounded control input constraints, without requiring prior knowledge of the bounds of the system parameters, nor the control direction. Assuming that the system is at-worst marginally stable, mean square boundedness of the closed-loop system states is proven. Lastly, numerical examples are presented to illustrate our results.
In this paper, we study a class of linear-quadratic (LQ) mean field games of controls with common noises and their corresponding $N$-player games. The theory of mean field game of controls considers a class of mean field games where the interaction is via the joint law of both the state and control. By the stochastic maximum principle, we first analyze the limiting behavior of the representative player and obtain his/her optimal control in a feedback form with the given distributional flow of the population and its control. The mean field equilibrium is determined by the Nash certainty equival…
Northwestern University（西北大学）；Princeton University（普林斯顿大学）；Georgia Institute of Technology（佐治亚理工学院）
We study discrete-time mean-field Markov games with infinite numbers of agents where each agent aims to minimize its ergodic cost. We consider the setting where the agents have identical linear state transitions and quadratic cost functions, while the aggregated effect of the agents is captured by the population mean of their states, namely, the mean-field state. For such a game, based on the Nash certainty equivalence principle, we provide sufficient conditions for the existence and uniqueness of its Nash equilibrium. Moreover, to find the Nash equilibrium, we propose a mean-field actor-criti…
