We consider optimal control of an unknown multi-agent linear quadratic (LQ) system where the dynamics and the cost are coupled across the agents through the mean-field (i.e., empirical mean) of the states and controls. Directly using single-agent LQ learning algorithms in such models results in regret which increases polynomially with the number of agents. We propose a new Thompson sampling based learning algorithm which exploits the structure of the system model and show that the expected Bayesian regret of our proposed algorithm for a system with agents of $|M|$ different types at time horizon $T$ is $\tilde{\mathcal{O}} \big( |M|^{1.5} \sqrt{T} \big)$ irrespective of the total number of agents, where the $\tilde{\mathcal{O}}$ notation hides logarithmic factors in $T$. We present detailed numerical experiments to illustrate the salient features of the proposed algorithm.
Approximate Thompson Sampling for Learning Linear Quadratic Regulators with $O(\sqrt{T})$ Regret
Department of Applied Artificial Intelligence, Seoul National University of Science and Technology（应用人工智能系，首尔科学技术大学）；Department of Electrical and Computer Engineering and ASRI, Seoul National University（电气电子工程系和ASRI，首尔国立大学）
Scalable regret for learning to control network-coupled subsystems with unknown dynamics
University of Southern California（南加州大学）；McGill University（麦吉尔大学）；Preferred Networks America
研究了由多个子系统通过网络连接而成的未知线性二次高斯（LQG）系统的控制问题。我们的目标是最小化和量化的遗憾（即性能损失），我们的战略相对于一个先知谁知道系统模型。全局地观察互连的子系统并直接使用全局系统的现有LQG学习算法会导致随着子系统的数量超线性地增加的遗憾。相反，我们提出了一个新的汤普森采样为基础的学习算法，利用底层网络的结构。我们表明，所提出的算法的预期遗憾是有界的$\tilde{\mathcal{O}} \big( n \sqrt{T} \big)$其中$n$是子系统的数量，$T$是时间范围和$\tilde{\mathcal{O}}(\cdot)$符号隐藏在$n$和$T$的对数项。因此，后悔与子系统的数量呈线性关系。我们提出了数值实验来说明所提出的算法的显着特点。
Policy Gradient with Self-Attention for Model-Free Distributed Nonlinear Multi-Agent Games
Department of Computer Science and Technology, University of Cambridge（计算机科学与技术系，剑桥大学）；Department of Electrical and Computer Engineering, University of California San Diego（电气与计算机工程系，加州大学圣地亚哥分校）；RoPeRt group, at DIIS - I3A, Universidad de Zaragoza（RoPeRt组，DIIS - I3A，阿拉贡大学）
