本文研究鲁棒协作式多智能体强化学习（RL）问题：大量具备分布式信息的协作智能体需在分布已知的随机不确定性与分布未知的非随机不确定性共存的场景下学习策略。针对同时考虑两类不确定性的策略优化问题，我们将其构建为最坏情况（极小极大）框架，该框架通常难以直接求解。因此我们聚焦线性二次场景以推导基准解。首先，由于分布式信息结构导致该问题尚无标准理论支撑，我们采用平均场型博弈（Mean-Field Type Game, MFTG）范式，从MFTG纳什均衡达成的角度为解的质量提供保证，进而可将其性能与对应的原始鲁棒多智能体控制问题进行对比。随后，我们提出一种滚动时域梯度下降上升强化学习算法以求解MFTG纳什均衡，并证明了其非渐近收敛速率。最后，我们通过数值实验验证了所提方法相较于基线算法的有效性。
英文摘要：
In this paper, we study the problem of robust cooperative multi-agent reinforcement learning (RL) where a large number of cooperative agents with distributed information aim to learn policies in the presence of \emph{stochastic} and \emph{non-stochastic} uncertainties whose distributions are respectively known and unknown. Focusing on policy optimization that accounts for both types of uncertainties, we formulate the problem in a worst-case (minimax) framework, which is is intractable in general. Thus, we focus on the Linear Quadratic setting to derive benchmark solutions. First, since no standard theory exists for this problem due to the distributed information structure, we utilize the Mean-Field Type Game (MFTG) paradigm to establish guarantees on the solution quality in the sense of achieved Nash equilibrium of the MFTG. This in turn allows us to compare the performance against the corresponding original robust multi-agent control problem. Then, we propose a Receding-horizon Gradient Descent Ascent RL algorithm to find the MFTG Nash equilibrium and we prove a non-asymptotic rate of convergence. Finally, we provide numerical experiments to demonstrate the efficacy of our approach relative to a baseline algorithm.
Exit-and-Join Dynamics and Equilibrium in Continuum Cooperative Games
Department of Electrical and Computer Engineering（电气与计算机工程系）；New York University Tandon School of Engineering（纽约大学塔恩工程学院）；Brooklyn, NY, USA（布鲁克林，纽约，美国）
Consensus Learning for Cooperative Multi-Agent Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Cooperative Game-Theoretic Credit Assignment for Multi-Agent Policy Gradients via the Core
Key Laboratory of Intelligence, Games and Information Processing in Shaanxi Province（陕西省智能信息处理重点实验室）；Unmanned System Research Institute（无人系统研究院）；Northwestern Polytechnical University（西北工业大学）；School of Mathematics and Statistics（数学与统计学院）；Information Science Academy（信息科学学院）；China Electronics Technology Group Corporation（中国电子科技集团有限公司）
