Learning in Mean Field Games: the Fictitious Play
Pierre Cardaliaguet, Saeed Hadikhanloo
平均场博弈系统描述了具有无穷多个无穷小相互作用主体的微分对策中的均衡配置。我们介绍了这些游戏的学习过程（类似于虚拟游戏），并显示其收敛时，平均场游戏是潜在的。
Mean Field Game systems describe equilibrium configurations in differential games with infinitely many infinitesimal interacting agents. We introduce a learning procedure (similar to the Fictitious Play) for these games and show its convergence when the Mean Field Game is potential.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
In this paper, a Nash-type fictitious game framework is introduced to handle a time-inconsistent linear-quadratic optimal control. The Nash-type game in this framework is called fictitious as it is between the decision maker (called real player) and an auxiliary control variable (called fictitious player) with the real player and fictitious player looking for time-consistent policy and precomitted optimal policy, respectively. Namely, the fictitious-game framework is actually an auxiliary-variable-based mechanism where the fictitious player is our particular design. Noting that the real player…
In this article we consider finite Mean Field Games (MFGs), i.e. with finite time and finite states. We adopt the framework introduced in Gomes Mohr and Souza in 2010, and study two seemly unexplored subjects. In the first one, we analyze the convergence of the fictitious play learning procedure, inspired by the results in continuous MFGs. In the second one, we consider the relation of some finite MFGs and continuous first order MFGs. Namely, given a continuous first order MFG problem and a sequence of refined space/time grids, we construct a sequence finite MFGs whose solutions admit limits p…
We develop the fictitious play algorithm in the context of the linear programming approach for mean field games of optimal stopping and mean field games with regular control and absorption. This algorithm allows to approximate the mean field game population dynamics without computing the value function by solving linear programming problems associated with the distributions of the players still in the game and their stopping times/controls. We show the convergence of the algorithm using the topology of convergence in measure in the space of subprobability measures, which is needed to deal with…
Convergence Analysis and Acceleration of Fictitious Play for General Mean-Field Games via the Best Response
本文研究一般平均场博弈中虚拟博弈算法的收敛性，通过分析最佳响应映射首次给出显式收敛速率，并提出基于回溯线搜索和分层网格的两种加速策略。
On the convergence of fictitious play algorithm in repeated games via the geometrical approach
本文利用几何方法识别出一类新的具有虚拟博弈收敛性（FPP）的3×3博弈，通过设计投影映射将高维系统约化为平面系统，并重新定义鞍点和汇点纳什均衡以证明连续时间虚拟博弈的收敛性。
On the Exponential Rate of Convergence of Fictitious Play in Potential Games
本文研究了连续时间下的虚拟游戏学习动力学。结果表明，在几乎每一个潜在的游戏，几乎每一个初始条件，FP的收敛速度是指数。特别是，本文重点研究的行为FP在潜在的游戏，其中所有的均衡的游戏是经常的，介绍了海萨尼。这种博弈被称为常规势博弈。最近它已被证明，几乎所有的潜在游戏（在勒贝格措施的意义上）是定期的。在本文中，它表明，在任何经常潜在的游戏（因此，在几乎每一个潜在的游戏），FP收敛到纳什均衡的集合在一个指数的速度从几乎每一个初始条件。
In this paper, we deepen the analysis of continuous time Fictitious Play learning algorithm to the consideration of various finite state Mean Field Game settings (finite horizon, $γ$-discounted), allowing in particular for the introduction of an additional common noise. We first present a theoretical convergence analysis of the continuous time Fictitious Play process and prove that the induced exploitability decreases at a rate $O(\frac{1}{t})$. Such analysis emphasizes the use of exploitability as a relevant metric for evaluating the convergence towards a Nash equilibrium in the context of Me…
In this paper, we apply the idea of fictitious play to design deep neural networks (DNNs), and develop deep learning theory and algorithms for computing the Nash equilibrium of asymmetric $N$-player non-zero-sum stochastic differential games, for which we refer as \emph{deep fictitious play}, a multi-stage learning process. Specifically at each stage, we propose the strategy of letting individual player optimize her own payoff subject to the other players' previous actions, equivalent to solve $N$ decoupled stochastic control optimization problems, which are approximated by DNNs. Therefore, th…
Robustness Properties in Fictitious-Play-Type Algorithms
模拟博弈（FP）是一种典型的博弈论学习算法，已广泛应用于分散控制场景。然而，FP和许多其他博弈论模型的标准处理，假设相当理想化的条件下，很少在现实的控制方案。本文考虑了一类广泛的最佳响应学习算法，我们称之为FP型算法。在这样的算法中，给定一些（可能有限的）关于行动历史的信息，每个人预测未来的行动，并根据他们的预测选择（短视的）最佳行动。我们提供了一个统一的分析FP型算法的行为下的一类重要的扰动，从而证明了鲁棒性的偏差，从理想的操作条件下，已经假定。这个鲁棒性的结果，然后使用两个控制相关的放松标准博弈论应用程序的收敛结果：分布式（基于网络的）实现没有完全的可观测性和异步部署（包括在连续时间）。在每种情况下，结果都是主要耐用性结果的直接结果。
Multi-agent learning using Fictitious Play and Extended Kalman Filter
分散优化任务是多智能体系统的重要组成部分。这些任务可以被解释为n-玩家潜在的游戏：因此，博弈论的学习算法可以用来解决分散的优化任务。虚构游戏是这些算法的典型例子。然而，虚拟博弈隐含地假设参与者有固定的策略。我们提出了一个新的变种，虚构的发挥球员预测他们的对手的战略，使用扩展卡尔曼滤波器，并使用他们的预测，以更新他们的战略。我们表明，在2 × 2的游戏，至少有一个纯纳什均衡和潜在的游戏中，玩家有两个可用的行动，该算法收敛到纯纳什均衡。该算法的性能进行了实证测试，在两个战略形式的游戏和一个ad-hoc传感器网络监控问题。该算法在这些游戏中的表现优于经典的虚拟游戏算法，因此提高了分散优化中的游戏理论学习的性能。
On the Global Convergence of Stochastic Fictitious Play in Stochastic Games with Turn-based Controllers
本文针对带回合式控制器的随机博弈，结合经典随机虚拟博弈与值迭代提出具备几乎必然收敛性的学习动力学，揭示了有限 horizon 长度递增与回合式控制器保证辅助博弈策略等价性的关键作用。
We study the convergence properties of decentralized fictitious play (DFP) for the class of near-potential games where the incentives of agents are nearly aligned with a potential function. In DFP, agents share information only with their current neighbors in a sequence of time-varying networks, keep estimates of other agents' empirical frequencies, and take actions to maximize their expected utility functions computed with respect to the estimated empirical frequencies. We show that empirical frequencies of actions converge to a set of strategies with potential function values that are larger…
Stochastic differential games have been used extensively to model agents' competitions in Finance, for instance, in P2P lending platforms from the Fintech industry, the banking system for systemic risk, and insurance markets. The recently proposed machine learning algorithm, deep fictitious play, provides a novel efficient tool for finding Markovian Nash equilibrium of large $N$-player asymmetric stochastic differential games [J. Han and R. Hu, Mathematical and Scientific Machine Learning Conference, pages 221-245, PMLR, 2020]. By incorporating the idea of fictitious play, the algorithm decoup…
Convergence of fictitious play for fully coupled FBSDEs in finite-player stochastic differential games
研究有限参与者随机微分博弈中完全耦合正倒向随机微分方程的虚拟策略收敛性，在一定假设下证明其收敛为几何收敛，特殊博弈中为超指数收敛，通过线性二次型银行间借贷问题数值实验予以证实。
On Robustness Properties in Empirical Centroid Fictitious Play
经验质心虚拟游戏（ECFP）是著名的虚拟游戏（FP）算法的推广，设计用于大型游戏中的实现。在ECFP中，一组局中人被细分为等价类，同一类中的局中人具有相似的属性。玩家通过跟踪和响应与每个等价类相关的汇总统计数据来选择下一阶段的行动。这种设置简化了跟踪和响应每个玩家的统计行为的艰巨任务，就像传统FP中的情况一样。除了ECFP之外，已经对经典FP提出了许多有用的修改，例如，规则允许基于网络的实现、增加的计算效率和更强的学习形式。这样的修改往往是很大的实用价值，但是，它们的有效性在很大程度上依赖于两个基本属性的FP：在经验分布步长过程中的变化的鲁棒性，和鲁棒性的最佳响应扰动。本文的主要贡献是表明，类似的鲁棒性也适用于ECFP算法。这一结果作为第一步，使实际修改ECFP，类似于那些已经开发的FP。
Learning controls in mean field games: the fictitious play and related gradient descent
本文研究势平均场博弈中两种学习过程：基于控制更新的虚拟博弈和主代理的梯度下降，均证明在一般条件下收敛到均衡解。
Consistency of vanishing smooth fictitious play
本文讨论了对策论中的一种策略--消失光滑虚拟博弈的一致性，它可以被看作是一个光滑的虚拟博弈过程，其中平滑参数是时间依赖的，并且是渐近消失的.这回答了Drew Fudenberg和Satoru Takahashi最初提出的一个问题。
Fictitious Play in Extensive-Form Games of Imperfect Information
研究不完全信息扩展式博弈中虚构博弈的长期行为，推广Monderer-Shapley收敛结果，并借助惯性和遗忘记忆证明实际博弈路径收敛到纯纳什均衡。
We propose a deep neural network-based algorithm to identify the Markovian Nash equilibrium of general large $N$-player stochastic differential games. Following the idea of fictitious play, we recast the $N$-player game into $N$ decoupled decision problems (one for each player) and solve them iteratively. The individual decision problem is characterized by a semilinear Hamilton-Jacobi-Bellman equation, to solve which we employ the recently developed deep BSDE method. The resulted algorithm can solve large $N$-player games for which conventional numerical methods would suffer from the curse of …
Learning in anonymous nonatomic games with applications to first-order mean field games
我们介绍了一个匿名游戏的模型与球员依赖的行动集。我们提出了几个学习过程的基础上著名的Fictionary Play和在线镜像下降，并证明了他们的收敛到经典的单调性条件下的平衡。典型的例子是一阶平均场博弈。
A fictitious-play finite-difference method for linearly solvable mean field games
针对含平流项多维系统导出的线性可解 MFGs，提出结合 Cole-Hopf 变换与虚拟博弈的迭代有限差分方法，证明其收敛性，并通过一维和二维数值实验验证稳定性。
Self-fictitious-play for Potential Monotone Ergodic Mean-field Games
针对环面上的遍历单调势平均场博弈，提出自虚构博弈动力学，证明其压缩性与唯一不变律，该不变律接近纳什均衡，误差为信念更新率平方根，线性二次示例与数值实验验证结果。
Empirical Centroid Fictitious Play: An Approach For Distributed Learning In Multi-Agent Games
本文提出经验质心虚构博弈算法，用于解决大规模多智能体游戏中的分布式学习问题，证明了该算法在潜在游戏中的收敛性，且适用于分布式环境。
Existing deep learning methods for solving mean-field games (MFGs) with common noise fix the sampling common noise paths and then solve the corresponding MFGs. This leads to a nested-loop structure with millions of simulations of common noise paths in order to produce accurate solutions, which results in prohibitive computational cost and limits the applications to a large extent. In this paper, based on the rough path theory, we propose a novel single-loop algorithm, named signatured deep fictitious play, by which we can work with the unfixed common noise setup to avoid the nested-loop struct…
On the Role of Tie-Breaking Rules in the Convergence of Fictitious Play for Symmetric First-Price Auctions
本研究探讨平局打破规则对虚拟博弈收敛性的影响，发现标准规则导致不收敛，而修改规则可恢复收敛并达到ε-均衡。
Entropic Fictitious Play for Mean Field Optimization Problem
针对两层神经网络的平均场优化问题，提出具有双循环结构的熵虚拟博弈算法以求解不动点，并证明其指数收敛性。
We analyze the convergence properties of the two-timescale fictitious play combining the classical fictitious play with the Q-learning for two-player zero-sum stochastic games with player-dependent learning rates. We show its almost sure convergence under the standard assumptions in two-timescale stochastic approximation methods when the discount factor is less than the product of the ratios of player-dependent step sizes. To this end, we formulate a novel Lyapunov function formulation and present a one-sided asynchronous convergence result.
We propose a distributed algorithm for multiagent systems that aim to optimize a common objective when agents differ in their estimates of the objective-relevant state of the environment. Each agent keeps an estimate of the environment and a model of the behavior of other agents. The model of other agents' behavior assumes agents choose their actions randomly based on a stationary distribution determined by the empirical frequencies of past actions. At each step, each agent takes the action that maximizes its expectation of the common objective computed with respect to its estimate of the envi…
A Computationally Efficient Implementation of Fictitious Play for Large-Scale Games
本文关注的是大规模环境下的分布式学习和优化。著名的模拟游戏（FP）算法已被证明可以在某些类别的多智能体游戏中实现纳什均衡学习。然而，当玩家数量很大时，FP在计算上可能难以实现。采样FP是FP的变体，其通过使用蒙特卡罗（即，基于抽样的方法。采样FP算法已经被研究作为分布式学习的工具和大规模问题的优化启发式。尽管采样FP具有计算优势，但其缺点是必须在算法的每一轮中抽取样本个数无限制地增长（大约为$\sqrt{t}$，其中$t$是重复播放的一轮）。在本文中，我们提出了计算效率采样FP（CESFP）--一个变种的采样FP中，只有一个样本需要提请每轮的算法（从$O(\sqrt{t})$样本每轮大幅减少，在采样FP的要求）。CESFP使用随机近似型规则来估计轮与轮之间的预期效用。它被证明是CESFP算法实现纳什均衡学习在相同意义上的经典FP和采样FP。仿真结果表明，CESFP的收敛速度（在重复播放迭代方面）与采样FP的收敛速度相似。
Approximation of deterministic mean field games under polynomial growth conditions on the data
针对动力学仿射于控制、代价泛函关于状态多项式增长的确定性平均场博弈，构造离散时间有限状态空间的逼近问题以求解原博弈均衡，并给出两个用虚拟博弈法求解的数值算例。
Generalized conditional gradient and learning in potential mean field games
本文将广义条件梯度算法用于二阶势单调平均场博弈，证明其等价于虚拟博弈，并给出多种步长下最优性差距、可利用性和解距离的收敛速率，线搜索步长可实现线性收敛。
On Randomized Fictitious Play for Approximating Saddle Points Over Convex Sets
给定两个由隶属度预言器指定的有界凸集$X\subseteq\RR^m$和$Y\subseteq\RR^n,$，以及一个连续的凸凹函数$F:X\times Y\to\RR$，我们考虑计算一个$\eps$ -近似鞍点，即一对$(x^*,y^*)\in X\times Y$的问题，使得$\sup_{y\in Y} F(x^*,y)\le \inf_{x\in X}F(x,y^*)+\eps.$ Grigoriadis和Khachiyan（1995）给出了一个计算矩阵对策的$\eps$ -近似鞍点的虚拟对策的简单随机变体，即，当$F$是双线性的且集合$X$和$Y$是单形时。在本文中，我们将他们的方法推广到一般情况。特别是，我们表明，对于恒定的“宽度”的功能，$\eps$近似鞍点可以计算使用$O^*(\frac{(n+m)}{\eps^2}\ln R)$随机样本从对数凹分布在凸集$X$和$Y$。假设$X$和$Y$具有半径为$1/R$的内接球和半径为$R$的外接球。因此，我们得到一个简单的随机多项式时间算法，计算这样的近似速度比已知的方法有界宽度的问题时，$\eps \in (0,1)$是一个固定的，但任意小的常数。我们实现这一结果的主要工具是随机虚构的发挥最近开发的结果从凸集抽样相结合。
From Weak Learning to Strong Learning in Fictitious Play Type Algorithms
本文研究了高度原型的Fictionary Play（FP）算法，以及广泛的一类基于最佳响应动态的学习过程，我们称之为FP型算法。FP的一个众所周知的缺点是，虽然参与者可以学习某种抽象意义上的均衡策略，但不能保证算法生成的逐期策略实际上收敛到均衡本身。这个问题从根本上与最佳响应对应的不连续性质有关，并且被许多FP型算法继承。它不仅在解释这些算法作为经济和社会学习的机制方面造成问题，而且还大大降低了这些算法在分布式控制中的实用价值。我们把参与者只在某种抽象意义上学习均衡的学习形式（将在本文中更精确地定义）称为弱学习，我们把参与者逐阶段策略收敛到均衡的学习形式称为强学习。提出了一种修改实现弱学习的FP型算法以构造实现强学习的变体的方法。理论上的收敛性结果得到了证明。
The entropic fictitious play (EFP) is a recently proposed algorithm that minimizes the sum of a convex functional and entropy in the space of measures -- such an objective naturally arises in the optimization of a two-layer neural network in the mean-field regime. In this work, we provide a concise primal-dual analysis of EFP in the setting where the learning problem exhibits a finite-sum structure. We establish quantitative global convergence guarantees for both the continuous-time and discrete-time dynamics based on properties of a proximal Gibbs measure introduced in Nitanda et al. (2022). …
Entropy Regularization in Mean-Field Games of Optimal Stopping
本文通过熵正则化将最优停止平均场博弈转化为奇异随机控制平均场博弈，建立均衡存在性与稳定性，并提出可收敛的虚拟对局算法，为无模型学习求解OS-MFG奠定理论基础。
Learning optimal policies in potential Mean Field Games: Smoothed Policy Iteration algorithms
本文提出两种平滑策略迭代算法，用于求解二阶势平均场博弈的纳什均衡，并在Lasry Lions单调性条件下证明全局收敛性，同时揭示其与虚拟对局算法的联系。
Extragradient methods for mean field games of controls and mean field type FBSDEs
提出一种基于超梯度方法的数值方案，用于求解由单调向量场驱动的耦合平均场正倒向随机微分方程，并证明在强单调性假设下近似解指数收敛。
We consider a dynamic traffic routing game over an urban road network involving a large number of drivers in which each driver selecting a particular route is subject to a penalty that is affine in the logarithm of the number of drivers selecting the same route. We show that the mean-field approximation of such a game leads to the so-called linearly solvable Markov decision process, implying that its mean-field equilibrium (MFE) can be found simply by solving a finite-dimensional linear system backward in time. Based on this backward-only characterization, it is further shown that the obtained…
Initialization-driven neural generation and training for high-dimensional optimal control and first-order mean field games
本文提出初始化驱动的神经网络方法，结合虚拟行动算法，实现高维最优控制与一阶平均场博弈的值函数及均衡的高效近似求解。
Many efficient algorithms have been designed to recover Nash equilibria of various classes of finite games. Special classes of continuous games with infinite strategy spaces, such as polynomial games, can be solved by semidefinite programming. In general, however, continuous games are not directly amenable to computational procedures. In this contribution, we develop an iterative strategy generation technique for finding a Nash equilibrium in a whole class of continuous two-person zero-sum games with compact strategy sets. The procedure, which is called the double oracle algorithm, has been su…
Learning by experience in Multi-Agent Systems (MAS) is a difficult and exciting task, due to the lack of stationarity of the environment, whose dynamics evolves as the population learns. In order to design scalable algorithms for systems with a large population of interacting agents (e.g. swarms), this paper focuses on Mean Field MAS, where the number of agents is asymptotically infinite. Recently, a very active burgeoning field studies the effects of diverse reinforcement learning algorithms for agents with no prior information on a stationary Mean Field Game (MFG) and learn their policy thro…
Learning Discrete-Time Major-Minor Mean Field Games
针对标准平均场博弈无法建模强影响力主要玩家的局限，提出离散时间主从平均场博弈（M3FGs）及基于虚拟博弈和概率单纯形划分的学习算法，通过理论和实验验证了模型适定性与算法收敛性，拓展了可处理博弈的类别。
We examine the long-run behavior of a wide range of dynamics for learning in nonatomic games, in both discrete and continuous time. The class of dynamics under consideration includes fictitious play and its regularized variants, the best-reply dynamics (again, possibly regularized), as well as the dynamics of dual averaging / "follow the regularized leader" (which themselves include as special cases the replicator dynamics and Friedman's projection dynamics). Our analysis concerns both the actual trajectory of play and its time-average, and we cover potential and monotone games, as well as gam…
Exploration noise for learning linear-quadratic mean field games
研究表明，在线性二次平均场博弈中，适当形式的公共噪声可充当探索噪声，并在无势或单调结构条件下保证虚拟博弈算法收敛。
Quadratic Programming Approach for Nash Equilibrium Computation in Multiplayer Imperfect-Information Games
针对多人非完美信息博弈，提出基于序列形式非线性互补问题的二次约束规划方法，利用非凸二次规划求解器精确计算纳什均衡，在三玩家Kuhn扑克中优于Gambit的logit方法。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。