arXiv 2103.006232021-03cs.AI

Scaling up Mean Field Games with Online Mirror Descent

DeepMind（深度思维）；Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Princeton University（普林斯顿大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Google Research（谷歌研究院）

Julien Perolat, Sarah Perrin, Romuald Elie, Mathieu Laurière, Georgios Piliouras, Matthieu Geist, Karl Tuyls, Olivier Pietquin

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

We address scaling up equilibrium computation in Mean Field Games (MFGs) using Online Mirror Descent (OMD). We show that continuous-time OMD provably converges to a Nash equilibrium under a natural and well-motivated set of monotonicity assumptions. This theoretical result nicely extends to multi-population games and to settings involving common noise. A thorough experimental investigation on various single and multi-population MFGs shows that OMD outperforms traditional algorithms such as Fictitious Play (FP). We empirically show that OMD scales up and converges significantly faster than FP by solving, for the first time to our knowledge, examples of MFGs with hundreds of billions states. This study establishes the state-of-the-art for learning in large-scale multi-agent and multi-population games.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

通过镜像下降策略优化实现异构多智能体强化学习

Heterogeneous Multi-Agent Reinforcement Learning via Mirror Descent Policy Optimization

Tarbiat Modares University（塔比阿特莫达勒斯大学）

针对异构协作 MARL 中智能体能力与策略不同的问题，提出 HAMDPO，利用多智能体优势分解和信赖域近似更新实现稳定单调改进，并在 MuJoCo、StarCraftII 上优于 HATRPO 与 HAPPO。

2023-08-13cs.AIcs.LGcs.MA

一种基于粒子的算法：通过变分传输与镜像下降在约束域上进行分布优化

A Particle-Based Algorithm for Distributional Optimization on \textit{Constrained Domains} via Variational Transport and Mirror Descent

University of Tsukuba（筑波大学）

本文提出镜像变分传输（mirrorVT）算法，通过镜像映射将约束域上的分布优化问题转换到无约束对偶域进行Wasserstein梯度下降，并验证了其在单纯形和欧几里得球约束域上的有效性与收敛性。

2022-08-03cs.AImath.OC

Variance Reduction on General Adaptive Stochastic Mirror Descent

Purdue University（普渡大学）

In this work, we investigate the idea of variance reduction by studying its properties with general adaptive mirror descent algorithms in nonsmooth nonconvex finite-sum optimization problems. We propose a simple yet generalized framework for variance reduced adaptive mirror descent algorithms named SVRAMD and provide its convergence analysis in both the nonsmooth nonconvex problem and the P-L conditioned problem. We prove that variance reduction reduces the SFO complexity of adaptive mirror descent algorithms and thus accelerates their convergence. In particular, our general theory implies tha…

2021-08-29cs.AIcs.LGmath.OC

超越状态级镜像下降：带有参数策略的离线策略优化

Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies

Nanjing University（南京大学）；UIUC（伊利诺伊大学）

本文研究了在一般函数逼近下离线强化学习的理论方面，扩展了理论保证到大或连续动作空间上的参数策略类，并提出了基于上下文耦合的新型分析和算法见解。

2026-10-12cs.AIcs.LG

随机均值流策略：带有熵镜降的一步生成控制

Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent

Laboratory for Big Data and Decision（大数据与决策实验室）；National University of Defense Technology（国防科技大学）；Samsung AI Center Cambridge（三星AI研究中心）；Queen Mary University of London（伦敦玛丽女王大学）；Fudan University（复旦大学）；ShanghaiTech University（上海科技大学）；Shanghai Innovation Institute（上海创新研究院）

本文提出了一种随机均值流策略（SMFP），通过均值流变换将高斯噪声映射到动作，以实现可训练的生成策略，从而在离线策略镜降框架下实现探索性且稳定的改进。

2026-10-12cs.AIcs.LG

Active Model Aggregation via Stochastic Mirror Descent

Georgia Institute of Technology（佐治亚理工学院）；Wisconsin Institutes for Discovery（威斯康星发现研究院）

We consider the problem of learning convex aggregation of models, that is as good as the best convex aggregation, for the binary classification problem. Working in the stream based active learning setting, where the active learner has to make a decision on-the-fly, if it wants to query for the label of the point currently seen in the stream, we propose a stochastic-mirror descent algorithm, called SMD-AMA, with entropy regularization. We establish an excess risk bounds for the loss of the convex aggregate returned by SMD-AMA to be of the order of $O\left(\sqrt{\frac{\log(M)}{T^{1-μ}}}\right)$,…

2015-03-28cs.AIcs.LGstat.ML

更多相关论文收起更多论文

Convergence of Online Mirror Descent

City University of Hong Kong（香港城市大学）

In this paper we consider online mirror descent (OMD) algorithms, a class of scalable online learning algorithms exploiting data geometric structures through mirror maps. Necessary and sufficient conditions are presented in terms of the step size sequence $\{η_t\}_{t}$ for the convergence of an OMD algorithm with respect to the expected Bregman distance induced by the mirror map. The condition is $\lim_{t\to\infty}η_t=0, \sum_{t=1}^{\infty}η_t=\infty$ in the case of positive variances. It is reduced to $\sum_{t=1}^{\infty}η_t=\infty$ in the case of zero variances for which the linear convergen…

2019-12-13cs.AIcs.LGmath.OC

策略镜像下降的功能加速

Functional Acceleration for Policy Mirror Descent

McGill University（麦吉尔大学）；Mila Quebec AI Institute（米拉魁北克人工智能研究所）；Google DeepMind（谷歌DeepMind）；CIFAR AI Chair（加拿大高级研究协会人工智能主席项目）

本文提出基于对偶性和功能加速的动量策略镜像下降更新，独立于参数化且适用于大规模优化，理论分析并数值验证了其动态特性及近似影响。

2025-03-25cs.AIcs.LGstat.ML

Mirror Descent Policy Optimization

University of Alberta（阿尔伯塔大学）；Technion（以色列理工学院）；Microsoft Research（微软研究院）；Google Research（谷歌研究院）

Mirror descent (MD), a well-known first-order method in constrained convex optimization, has recently been shown as an important tool to analyze trust-region algorithms in reinforcement learning (RL). However, there remains a considerable gap between such theoretically analyzed algorithms and the ones used in practice. Inspired by this, we propose an efficient RL algorithm, called {\em mirror descent policy optimization} (MDPO). MDPO iteratively updates the policy by {\em approximately} solving a trust-region problem, whose objective function consists of two terms: a linearization of the stand…

2021-06-07cs.AIcs.LGstat.ML

Policy Search using Dynamic Mirror Descent MPC for Model Free Off Policy RL

Indian Institute of Science（印度科学研究所）

Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches to get the best from both: asymptotic performance of Mf-RL and high sample-efficiency of Mb-RL. Inspired by these works, we propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL. In particular, two loops are proposed, where the Dynamic Mirror Descent based Model Predictive Control (DMD-MPC) is used as the inner loop to obtain an optimal sequence of actions. These actions are in turn used to sign…

2021-10-23cs.AIcs.LGcs.RO