arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.06845math.OCstat.ML
收藏

连续时间线性二次平均场控制/对策策略梯度的整体收敛

Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time

Weichen Wang, Jiequn Han, Zhuoran Yang, Zhaoran Wang

摘要:

强化学习是一种强大的工具,可以通过与环境的交互来学习可能多个代理的最优策略。当代理的数量增长到非常大时,系统可以近似为平均场问题。因此,它激发了新的研究方向的平均场控制(MFC)和平均场游戏(MFG)。本文研究了线性二次平均场控制与对策的策略梯度法,其中假设每个代理具有相同的线性状态转移和二次费用函数。虽然大多数最近的工作的政策梯度MFC和MFG是基于离散时间模型,我们专注于连续时间模型的一些分析技术可以是有趣的读者。对于MFC和MFG,我们提供了策略梯度更新,并表明它以线性速度收敛到最优解,这是通过合成模拟验证。对于MFG,我们也提供了纳什均衡存在唯一的充分条件。

英文摘要:

Reinforcement learning is a powerful tool to learn the optimal policy of possibly multiple agents by interacting with the environment. As the number of agents grow to be very large, the system can be approximated by a mean-field problem. Therefore, it has motivated new research directions for mean-field control (MFC) and mean-field game (MFG). In this paper, we study the policy gradient method for the linear-quadratic mean-field control and game, where we assume each agent has identical linear state transitions and quadratic cost functions. While most of the recent works on policy gradient for MFC and MFG are based on discrete-time models, we focus on the continuous-time models where some analyzing techniques can be interesting to the readers. For both MFC and MFG, we provide policy gradient update and show that it converges to the optimal solution at a linear rate, which is verified by a synthetic simulation. For MFG, we also provide sufficient conditions for the existence and uniqueness of the Nash equilibrium.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑