arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.04780math.OC
收藏

群体趋同平均场博弈中的学习:具有有限时间收敛性分析的单环算法

Learning in Herding Mean Field Games: Single-Loop Algorithm with Finite-Time Convergence Analysis

Sihan Zeng, Sujay Bhatt, Alec Koppel, Sumitra Ganesh

AI总结:

针对未知动力学离散时间静态MFG的均衡求解问题,提出“完全群体趋同类”MFG扩展可解范围,设计ASAC-MFG单环算法实现有限时间收敛,在MDP退化场景下样本复杂度达最优且无需Bellman算子收缩性假设。

摘要:

我们考虑具有未知动力学的离散时间静态平均场博弈(MFG),并设计具有有限时间复杂度保证的均衡求解算法。现有解决方案需假设平均场最优性一致性算子的收缩性或严格弱单调性,这些条件过于严苛。本研究提出一类新的可解MFG,称为“完全群体趋同类”,它扩展了已知的可解MFG类,首次包含具有多个均衡的问题。我们提出一种直接策略优化方法——平均场博弈加速单环演员-评论家算法(ASAC-MFG),在可获取单一马尔可夫样本轨迹的情况下,该算法可证明能为该类MFG找到全局均衡。与现有方法不同,ASAC-MFG是单环且基于单样本路径的。我们通过为多时间尺度随机近似开发的新技术,建立了ASAC-MFG收敛到平均场均衡的有限时间和有限样本保证。我们通过示例数值模拟验证了理论结果。当平均场不影响转移和奖励时,MFG退化为马尔可夫决策过程(MDP),ASAC-MFG成为求解平均奖励MDP最优策略的演员-评论家算法,其样本复杂度与当前最优水平相当。现有工作在假设Bellman算子收缩性的前提下推导复杂度,但该假设对平均奖励MDP不成立;我们通过改进的Lyapunov函数移除了这一不合理假设,同时达到了相同的复杂度水平。

英文摘要:

We consider discrete-time stationary mean field games (MFG) with unknown dynamics and design algorithms for finding the equilibrium with finite-time complexity guarantees. Prior solutions to the problem assume either the contraction of a mean field optimality-consistency operator or strict weak monotonicity, which may be overly restrictive. In this work, we introduce a new class of solvable MFGs, named the "fully herding class", which expands the known solvable class of MFGs and for the first time includes problems with multiple equilibria. We propose a direct policy optimization method, Accelerated Single-loop Actor Critic Algorithm for Mean Field Games (ASAC-MFG), that provably finds a global equilibrium for MFGs within this class, under suitable access to a single trajectory of Markovian samples. Different from the prior methods, ASAC-MFG is single-loop and single-sample-path. We establish the finite-time and finite-sample convergence of ASAC-MFG to a mean field equilibrium via new techniques that we develop for multi-time-scale stochastic approximation. We support the theoretical results with illustrative numerical simulations. When the mean field does not affect the transition and reward, a MFG reduces to a Markov decision process (MDP) and ASAC-MFG becomes an actor-critic algorithm for finding the optimal policy in average-reward MDPs, with a sample complexity matching the state-of-the-art. Previous works derive the complexity assuming a contraction on the Bellman operator, which is invalid for average-reward MDPs. We match the rate while removing the untenable assumption through an improved Lyapunov function.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑