We investigate team optimal control of stochastic subsystems that are weakly coupled in dynamics (through the mean-field of the system) and are arbitrary coupled in the cost. The controller of each subsystem observes its local state and the mean-field of the state of all subsystems. The system has a non-classical information structure. Exploiting the symmetry of the problem, we identify an information state and use that to obtain a dynamic programming decomposition. This dynamic program determines a globally optimal strategy for all controllers. Our solution approach works for arbitrary number of controllers and generalizes to the setup when the mean-field is observed with noise. The size of the information state is time-invariant; thus, the results generalize to the infinite-horizon control setups as well. In addition, when the mean-field is observed without noise, the size of the corresponding information state increases polynomially (rather than exponentially) with the number of controllers which allows us to solve problems with moderate number of controllers. We illustrate our approach by an example motivated by smart grids that consists of $100$ coupled subsystems.
In this paper, we investigate team optimal control of coupled major-minor subsystems with mean-field sharing. In such a model, there is one major subsystem that directly influences the dynamics of $n$ homogeneous minor subsystems; however, the minor subsystems influence the dynamics of the major subsystem and each other only through their mean behaviour (indirectly). In this model, the major and the minor subsystems are arbitrarily coupled in the cost and have mean-field sharing information structure. We propose a two-step approach. In the first step, we describe a mean-field sharing model wit…
A decentralized control system with linear dynamics, quadratic cost, and Gaussian disturbances is considered. The system consists of a finite number of subsystems whose dynamics and per-step cost function are coupled through their mean-field (empirical average). The system has mean-field sharing information structure, i.e., each controller observes the state of its local subsystem (either perfectly or with noise) and the mean-field. It is shown that the optimal control law is unique, linear, and identical across all subsystems. Moreover, the optimal gains are computed by solving two decoupled …
This paper investigates a soft-constrained MinMax control problem of a leader-follower network. The network consists of one leader and an arbitrary number of followers that wish to reach consensus with minimum energy consumption in the presence of external disturbances. The leader and followers are coupled in the dynamics and cost function. Two non-classical information structures are considered: mean-field sharing and intermittent mean-field sharing, where the mean-field refers to the aggregate state of the followers. In mean-field sharing, every follower observes its local state, the state o…
Optimality of Symmetric Independent Policies under Decentralized Mean-Field Information Sharing for Stochastic Teams and Equivalence with McKean-Vlasov Control of a Representative Agent
Royal Military College of Canada（加拿大皇家军事学院）；Bilkent University（比尔肯特大学）；Queen’s University（女王大学）
In stochastic optimal control, change of measure arguments have been crucial for stochastic analysis. Such an approach is often called static reduction in dynamic team theory (or decentralized stochastic control) and has been an effective method for establishing existence and approximation results for optimal policies. In this paper, we place such static reductions into three categories: (i) those that are policy-independent (as those introduced by Witsenhausen), (ii) those that are policy-dependent (as those introduced by Ho and Chu for partially nested dynamic teams), and (iii) those that we…
Motivated by the recent developments in artificial intelligence, we introduce linear quadratic deep structured teams in this paper. Two notions of equivariant and partially equivariant systems are defined, and it is shown that such systems can be partitioned into a few sub-populations of decision makers, where every decision maker in each sub-population is coupled in both dynamics and cost function through a set of linear regressions of the states and actions of all decision makers. Two non-classical information structures are considered: deep-state sharing and partial deep-state sharing, wher…
This paper considers the problem of decentralized analysis and control synthesis to verify and ensure properties like stability and dissipativity of a large-scale networked system comprised of linear subsystems interconnected in an arbitrary topology. In particular, we design systematic networked system analysis and control synthesis processes that can be executed in a decentralized manner at the subsystem level with minimal information sharing among the subsystems. Compared to our most recent work on the same topic, we consider a substantially more generalized problem setup in this paper and …
In this paper, we study a dynamic version of the sharing problem, in which a dynamic system cost function composed of time-variant local costs of subsystems and a shared time-variant cost of the whole system is minimized. A dynamic alternating direction method of multipliers (ADMM) is proposed to track the varying optimal points of the dynamic optimization problem in an online manner. We analyze the convergence properties of the dynamic ADMM and show that, under several standard technical assumptions, the iterations of the dynamic ADMM converge linearly to some neighborhoods of the time-varyin…
This paper studies a large number of homogeneous Markov decision processes where the transition probabilities and costs are coupled in the empirical distribution of states (also called mean-field). The state of each process is not known to others, which means that the information structure is fully decentralized. The objective is to minimize the average cost, defined as the empirical mean of individual costs, for which a sub-optimal solution is proposed. This solution does not depend on the number of processes, yet it converges to the optimal solution of the so-called mean-field sharing as the…
In this paper, we are interested in systems with multiple agents that wish to collaborate in order to accomplish a common task while a) agents have different information (decentralized information) and b) agents do not know the model of the system completely i.e., they may know the model partially or may not know it at all. The agents must learn the optimal strategies by interacting with their environment i.e., by decentralized Reinforcement Learning (RL). The presence of multiple agents with different information makes decentralized reinforcement learning conceptually more difficult than cent…
In this paper, we study distributed estimation and control problems over graphs under partially nested information patterns. We show a duality result that is very similar to the classical duality result between state estimation and state feedback control with a classical information pattern, under the condition that the disturbances entering different systems on the graph are uncorrelated. The distributed estimation problem decomposes into $N$ separate estimation problems, where $N$ is the number of interconnected subsystems over the graph, and the solution to each subproblem is simply the opt…
Isomorphism Properties of Optimality and Equilibrium Solutions under Equivalent Information Structure Transformations: Stochastic Dynamic Games and Teams
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Georgia Institute of Technology（佐治亚理工学院）；University of Nevada, Reno（内华达大学雷诺分校）
A distributed stochastic optimal control solution is presented for cooperative multi-agent systems. The network of agents is partitioned into multiple factorial subsystems, each of which consists of a central agent and neighboring agents. Local control actions that rely only on agents' local observations are designed to optimize the joint cost functions of subsystems. When solving for the local control actions, the joint optimality equation for each subsystem is cast as a linear partial differential equation and solved using the Feynman-Kac formula. The solution and the optimal control action …
In this paper we study a class of matrix-valued linear-quadratic mean-field-type games for both the risk-neutral, risk-sensitive and robust cases. Non-cooperation, full cooperation and adversarial between teams are treated. We provide a semi-explicit solution for both problems by means of a direct method. The state dynamics is described by a matrix-valued linear jump-diffusion-regime switching system of conditional mean-field type where the conditioning is with respect to common noise which is a regime switching process. The optimal strategies are in a state-and-conditional mean-field feedback…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Georgia Institute of Technology（佐治亚理工学院）；University of Nevada, Reno（内华达大学里诺分校）
Distributed algorithms for both discrete-time and continuous-time linearly solvable optimal control (LSOC) problems of networked multi-agent systems (MASs) are investigated in this paper. A distributed framework is proposed to partition the optimal control problem of a networked MAS into several local optimal control problems in factorial subsystems, such that each (central) agent behaves optimally to minimize the joint cost function of a subsystem that comprises a central agent and its neighboring agents, and the local control actions (policies) only rely on the knowledge of local observation…
