Motivated by recent developments in mean field games in ecology, in this paper we introduce a connection between the best response dynamics in evolutionary game theory, the minimization of the highest income of a game, and minimizing movement schemes. The aim of this work is to develop a variational approach to compute solutions of first order ergodic mean field games that may not possess a priori a variational structure. The study is complemented by a discussion and successful implementation of numerical algorithms, and comparisons between them in a variety of cases.
We present an example of symmetric ergodic $N$-players differential games, played in memory strategies on the position of the players, for which the limit set, as $N\to +\infty$, of Nash equilibrium payoffs is large, although the game has a single mean field game equilibrium. This example is in sharp contrast with a result by Lacker [23] for finite horizon problems.
We study some ergodicity property of zero-sum stochastic games with a finite state space and possibly unbounded payoffs. We formulate this property in operator-theoretical terms, involving the solvability of an optimality equation for the Shapley operators (i.e., the dynamic programming operators) of a family of perturbed games. The solvability of this equation entails the existence of the uniform value, and its solutions yield uniform optimal stationary strategies. We first provide an analytical characterization of this ergodicity property, and address the generic uniqueness, up to an additiv…
The ergodic equation is a basic tool in the study of mean-payoff stochastic games. Its solvability entails that the mean payoff is independent of the initial state. Moreover, optimal stationary strategies are readily obtained from its solution. In this paper, we give a general sufficient condition for the solvability of the ergodic equation, for a game with finite state space but arbitrary action spaces. This condition involves a pair of directed hypergraphs depending only on the ``growth at infinity'' of the Shapley operator of the game. This refines a recent result of the authors which only …
The goal of this paper is to study the long time behavior of solutions of the first-order mean field game (MFG) systems with a control on the acceleration. The main issue for this is the lack of small time controllability of the problem, which prevents to define the associated ergodic mean field game problem in the standard way. To overcome this issue, we first study the long-time average of optimal control problems with control on the acceleration: we prove that the time average of the value function converges to an ergodic constant and represent this ergodic constant as a minimum of a Lagran…
Mean payoff stochastic games can be studied by means of a nonlinear spectral problem involving the Shapley operator: the ergodic equation. A solution consists in a scalar, called the ergodic constant, and a vector, called bias. The existence of such a pair entails that the mean payoff per time unit is equal to the ergodic constant for any initial state, and the bias gives stationary strategies. By exploiting two fundamental properties of Shapley operators, monotonicity and additive homogeneity, we give a necessary and sufficient condition for the solvability of the ergodic equation for all the…
We study nonzero-sum stochastic games for continuous time Markov decision processes on a denumerable state space with risk-sensitive ergodic cost criterion. Transition rates and cost rates are allowed to be unbounded. Under a Lyapunov type stability assumption, we show that the corresponding system of coupled HJB equations admits a solution which leads to the existence of a Nash equilibrium in stationary strategies. We establish this using an approach involving principal eigenvalues associated with the HJB equations. Furthermore, exploiting appropriate stochastic representation of principal ei…
The infinite horizon risk-sensitive discounted-cost and ergodic-cost nonzero-sum stochastic games for controlled Markov chains with countably many states are analyzed. For the discounted-cost game, we prove the existence of Nash equilibrium strategies in the class of Markov strategies under fairly general conditions. Under an additional geometric ergodicity condition and a small cost criterion, the existence of Nash equilibrium strategies in the class of stationary Markov strategies is proved for the ergodic-cost game.
We consider mean field games with ergodic cost in the framework of a general discrete time controlled Markov processes. The state space of the processes is given by a general $σ$-compact Polish space. Under certain conditions, we show the existence of a mean field game equilibrium. We also study the $N$-person game where the players interacts with each other via their empirical measure. We show that the $N$-person game has Nash equilibrium and as $N$ tends to infinity the equilibria converge to a mean field game solution.
We study two person nonzero-sum stochastic differential games with risk-sensitive discounted and ergodic cost criteria. Under certain conditions we establish a Nash equilibrium in Markov strategies for the discounted cost criterion and a Nash equilibrium in stationary strategies for the ergodic cost criterion. We achieve our results by studying the relevant systems of coupled HJB equations.
A general class of mean field games are considered where the governing dynamics are controlled diffusions in $\mathbb{R}^d$. The optimization criterion is the long time average of a running cost function. Under various sets of hypotheses, we establish the existence of mean field game solutions. We also study the long time behavior of mean field game solutions associated with the finite horizon problem, and under the assumption of geometric ergodicity for the dynamics, we show that these converge to the ergodic mean field game solution as the horizon tends to infinity. Lastly, we study the asso…
Recently, Sidford, Wang, Wu and Ye (2018) developed an algorithm combining variance reduction techniques with value iteration to solve discounted Markov decision processes. This algorithm has a sublinear complexity when the discount factor is fixed. Here, we extend this approach to mean-payoff problems, including both Markov decision processes and perfect information zero-sum stochastic games. We obtain sublinear complexity bounds, assuming there is a distinguished state which is accessible from all initial states and for all policies. Our method is based on a reduction from the mean payoff pr…
We study infinite horizon discounted-cost and ergodic-cost risk-sensitive zero-sum stochastic games for controlled continuous time Markov chains on a countable state space. For the discounted-cost game we prove the existence of value and saddle-point equilibrium in the class of Markov strategies under nominal conditions. For the ergodic-cost game we prove the existence of values and saddle point equilibrium by studying the corresponding Hamilton-Jacobi-Isaacs equation under a certain Lyapunov condition.
A basic question for zero-sum repeated games consists in determining whether the mean payoff per time unit is independent of the initial state. In the special case of "zero-player" games, i.e., of Markov chains equipped with additive functionals, the answer is provided by the mean ergodic theorem. We generalize this result to repeated games. We show that the mean payoff is independent of the initial state for all state-dependent perturbations of the rewards if and only if an ergodicity condition is verified. The latter is characterized by the uniqueness modulo constants of nonlinear harmonic f…
Value existence for zero-sum ergodic stochastic differential games
School of Mathematics and Statistics, Shandong University（山东大学数学与统计学院）；Research Center for Mathematics and Interdisciplinary Sciences, Shandong University（山东大学数学交叉科学研究院）；Department of Mathematical Sciences, Durham University（杜伦大学数学科学系）
Mean field games model equilibria in games with a continuum of players as limiting systems of symmetric $n$-player games with weak interaction between the players. We consider a finite-state, infinite-horizon problem with two cost criteria: discounted and ergodic. Under the Lasry--Lions monotonicity condition we characterize the stationary ergodic mean field game equilibrium by a mean field game system of two coupled equations: one for the value and the other for the stationary measure. This system is linked with the ergodic master equation. Several discounted mean field game systems are utili…
We consider stochastic differential games with $N$ players, linear-Gaussian dynamics in arbitrary state-space dimension, and long-time-average cost with quadratic running cost. Admissible controls are feedbacks for which the system is ergodic. We first study the existence of affine Nash equilibria by means of an associated system of $N$ Hamilton-Jacobi-Bellman and $N$ Kolmogorov-Fokker-Planck partial differential equations. We give necessary and sufficient conditions for the existence and uniqueness of quadratic-Gaussian solutions in terms of the solvability of suitable algebraic Riccati and S…
Let $T:X\to X $ and $S:Y \to Y$ be continuous maps defined on compact sets. Let $$φ_i(μ,ν)=\int_{X \times Y} A_i(x,y) dμ(x) dν(y)\;\;{for} \;\; i=1,2,$$ where $μ$ is $T$-invariant and $ν$ is $S$-invariant, be pay-off functions for a game (in the usual sense of game theory) between players that have the set of invariant measures for $T$ (player 1) and $S$ (player 2) as possible strategies. Our goal here is to establish the notion of Nash equilibrium point for the game defined by this pay-offs and strategies. The main tools came from ergodic optimization (as we are optimizing over the set of inv…
We show that the long time average of solutions of first order mean field game systems in finite horizon is governed by an ergodic system of mean field game type. The well-posedness of this later system and the uniqueness of the ergodic constant rely on weak KAM theory.
We study nonzero-sum stochastic games for continuous time Markov chains on a denumerable state space with risk sensitive discounted and ergodic cost criteria. For the discounted cost criterion we first show that the corresponding system of coupled HJB equations has an appropriate solution. Then under an additional additive structure on the transition rate matrix and payoff functions, we establish the existence of a Nash equilibrium in Markov strategies. For the ergodic cost criterion we assume a Lyapunov type stability assumption and a small cost condition. Under these assumptions we show that…
The aim of this paper is to study the long time behavior of solutions to deterministic mean field games systems on Euclidean space. This problem was addressed on the torus ${\mathbb T}^n$ in [P. Cardaliaguet, {\it Long time average of first order mean field games and weak KAM theory}, Dyn. Games Appl. 3 (2013), 473-488], where solutions are shown to converge to the solution of a certain ergodic mean field games system on ${\mathbb T}^n$. By adapting the approach in [A. Fathi, E. Maderna, {\it Weak KAM theorem on non compact manifolds}, NoDEA Nonlinear Differential Equations Appl. 14 (2007), 1-…
In this paper we consider two-person zero-sum risk-sensitive stochastic dynamic games with Borel state and action spaces and bounded reward. The term risk-sensitive refers to the fact that instead of the usual risk neutral optimization criterion we consider the exponential certainty equivalent. The discounted reward case on a finite and an infinite time horizon is considered, as well as the ergodic reward case. Under continuity and compactness conditions we prove that the value of the game exists and solves the Shapley equation and we show the existence of optimal (non-stationary) strategies. …
We consider a system of mean field games with local coupling in the deterministic limit. Under general structure conditions on the Hamiltonian and coupling, we prove existence and uniqueness of the weak solution, characterizing this solution as the minimizer of some optimal control of Hamilton-Jacobi and continuity equations. We also prove that this solution converges in the long time average to the solution of the associated ergodic problem.
This paper investigates the optimal ergodic sublinear convergence rate of the relaxed proximal point algorithm for solving monotone variational inequality problems. The exact worst case convergence rate is computed using the performance estimation framework. It is observed that, as the number of iterations getting larger, this numerical rate asymptotically coincides with an existing sublinear rate, whose optimality is unknown. This hints that, without further assumptions, sublinear convergence rate is likely the best achievable rate. A concrete example is constructed, which provides a lower bo…
We propose two algorithms for the solution of the optimal control of ergodic McKean-Vlasov dynamics. Both algorithms are based on approximations of the theoretical solutions by neural networks, the latter being characterized by their architecture and a set of parameters. This allows the use of modern machine learning tools, and efficient implementations of stochastic gradient descent.The first algorithm is based on the idiosyncrasies of the ergodic optimal control problem. We provide a mathematical proof of the convergence of the approximation scheme, and we analyze rigorously the approximatio…
In this paper, we introduce an abstract fuzzy economy (generalized fuzzy game) model with a countable space of actions and we study the existence of the fuzzy equilibrium. As applications, two types of results are obtained. The first ones concern the existence of the solutions for systems of generalized quasi-variational inequalities with random fuzzy mappings which we define. The last ones are new random fixed point theorems for correspondences with values in complete countable metric spaces.
We derive necessary optimality conditions for minimizers of regular functionals in the calculus of variations under smooth state constraints. In the literature, this classical problem is widely investigated. The novelty of our result lies in the fact that the presence of state constraints enters the Euler-Lagrange equations as a local feedback, which allows to derive the $C^{1,1}$-smoothness of solutions. As an application, we discuss a constrained Mean Field Games problem, for which our optimality conditions allow to construct Lipschitz relaxed solutions, thus improving the existence result i…
We consider stochastic differential games with $N$ nearly identical players, linear-Gaussian dynamics, and infinite horizon discounted quadratic cost. Admissible controls are feedbacks for which the system is ergodic. We first study the existence of affine Nash equilibria by means of an associated system of $N$ Hamilton-Jacobi-Bellman and $N$ Kolmogorov-Fokker-Planck partial differential equations, proving that for small discount factors quadratic-Gaussian solutions exist and are unique. Then, we prove the convergence of such solutions to the unique quadratic-Gaussian solution of the pair of M…
It is known that the generalized Nash equilibrium problem can be reformulated as a quasivariational inequality. Our aim in this work is to introduce a variational approach to study the existence of solutions for generalized ordinal Nash games, that is, generalized games where the player preferences are binary relations that do not necessarily admit a utility representation.
We explore a mechanism of decision-making in Mean Field Games with myopic players. At each instant, agents set a strategy which optimizes their expected future cost by assuming their environment as immutable. As the system evolves, the players observe the evolution of the system and adapt to their new environment without anticipating. With a specific cost structures, these models give rise to coupled systems of partial differential equations of quasi-stationary nature. We provide sufficient conditions for the existence and uniqueness of classical solutions for these systems, and give a rigorou…
In this article, we introduce a method to approximate solutions of some variational mean field game problems with congestion, by finite sets of player trajectories. These trajectories are obtained by solving a minimization problem similar to the initial variational problem. In this discretized problem, congestion is penalized by a Moreau envelop with the 2-Wasserstein distance. Study of this envelop as well as efficient computation of its values and variations is done using semi-discrete optimal transport. We show convergence of the discrete sets of trajectories toward a solution of the mean f…
We investigate the existence and uniqueness of (locally) absolutely continuous trajectories of a penalty term-based dynamical system associated to a constrained variational inequality expressed as a monotone inclusion problem. Relying on Lyapunov analysis and on the ergodic continuous version of the celebrated Opial Lemma we prove weak ergodic convergence of the orbits to a solution of the constrained variational inequality under investigation. If one of the operators involved satisfies stronger monotonicity properties, then strong convergence of the trajectories can be shown.
After defining a pure-action profile in a nonatomic aggregative game, where players have specific compact convex pure-action sets and nonsmooth convex cost functions, as a square-integrable function, we characterize a Wardrop equilibrium as a solution to an infinite-dimensional generalized variational inequality. We show the existence of Wardrop equilibrium and variational Wardrop equilibrium, a concept of equilibrium adapted to the presence of coupling constraints, in monotone nonatomic aggregative games. The uniqueness of (variational) Wardrop equilibrium is proved for strictly or aggregativ…
Motivated by applications in natural resource management, risk management, and finance, this paper is focused on an ergodic two-sided singular control problem for a general one-dimensional diffusion process. The control is given by a bounded variation process. Under some mild conditions, the optimal reward value as well as an optimal control policy are derived by the vanishing discount method. Moreover, the Abelian and Cesàro limits are established. Then a direct solution approach is provided at the end of the paper.
