Stochastic games are an important class of problems that generalize Markov decision processes to game theoretic scenarios. We consider finite state two-player zero-sum stochastic games over an infinite time horizon with discounted rewards. The players are assumed to have infinite strategy spaces and the payoffs are assumed to be polynomials. In this paper we restrict our attention to a special class of games for which the single-controller assumption holds. It is shown that minimax equilibria and optimal strategies for such games may be obtained via semidefinite programming.
Many important values for cooperative games are known to arise from least square optimization problems. The present investigation develops an optimization framework to explain and clarify this phenomenon in a general setting. The main result shows that every linear value results from some least square approximation problem and that, conversely, every least square approximation problem with linear constraints yields a linear value. This approach includes and extends previous results on so-called least square values and semivalues in the literature. In particular, is it demonstrated how known ex…
We consider a formulation of a non zero-sum n players game by an n+1 players zero-sum game. We suppose the existence of the n+1-th player in addition to n players in the main game, and virtual subsidies to the n players which is provided by the n+1-th player. Its strategic variable affects only the subsidies, and does not affect choice of strategies by the n players in the main game. His objective function is the opposite of the sum of the payoffs of the n players. We will show 1) The minimax theorem by Sion (Sion(1958)) implies the existence of Nash equilibrium in the n players non zero-sum g…
We give an example of a zero-sum stochastic game with four states, compact action sets for each player, and continuous payoff and transition functions, such that the discounted value does not converge as the discount factor tends to 0, and the value of the n-stage game does not converge as n goes to infinity.
Dynamic zero-sum games are an important class of problems with applications ranging from evasion-pursuit and heads-up poker to certain adversarial versions of control problems such as multi-armed bandit and multiclass queuing problems. These games are generally very difficult to solve even when one player's strategy is fixed, and so constructing and evaluating good sub-optimal policies for each player is an important practical problem. In this paper, we propose the use of information relaxations to construct dual lower and upper bounds on the optimal value of the game. We note that the informa…
A zero-sum two-person Perfect Information Semi-Markov game (PISMG) under limiting ratio average payoff has a value and both the maximiser and the minimiser have optimal pure semi-stationary strategies. We arrive at the result by first fixing an arbitrary initial state and forming the matrix of undiscounted payoffs corresponding to each pair of pure stationary strategies of the two players and proving that this matrix has a pure saddle point.
Machine learning processes, e.g. ''learning in games'', can be viewed as non-linear dynamical systems. In general, such systems exhibit a wide spectrum of behaviors, ranging from stability/recurrence to the undesirable phenomena of chaos (or ''butterfly effect''). Chaos captures sensitivity of round-off errors and can severely affect predictability and reproducibility of ML systems, but AI/ML community's understanding of it remains rudimentary. It has a lot out there that await exploration. Recently, Cheung and Piliouras employed volume-expansion argument to show that Lyapunov chaos occurs in …
The game of baccarat has evolved from a parlor game played by French aristocrats in the first half of the 19th century to a casino game that generated over US\$41 billion in revenue for the casinos of Macau in 2013. The parlor game was originally a three-person zero-sum game. Later in the 19th century it was simplified to a two-person zero-sum game. Early in the 20th century the parlor game became a casino game, no longer zero-sum. In the mid 20th century, the strategic casino game became a nonstrategic game, with players competing against the house instead of against each other. We argue that…
We present efficient algorithms for computing optimal or approximately optimal strategies in a zero-sum game for which Player I has n pure strategies and Player II has an arbitrary number of pure strategies. We assume that for any given mixed strategy of Player I, a best response or "approximate" best response of Player II can be found by an oracle in time polynomial in n. We then show how our algorithms may be applied to several search games with applications to security and counter-terrorism. We evaluate our main algorithm experimentally on a prototypical search game. Our results show it per…
This short note demonstrates how one can define a transformation of a non-zero sum game into a zero sum, so that the optimal mixed strategy achieving equilibrium always exists. The transformation is equivalent to introduction of a passive player into a game (a player with a singleton set of pure strategies), whose payoff depends on the actions of the active players, and it is justified by the law of conservation of utility in a game. In a transformed game, each participant plays against all other players, including the passive player. The advantage of this approach is that the transformed game…
We consider discrete time partially observable zero-sum stochastic game with average payoff criterion. We study the game using an equivalent completely observable game. We show that the game has a value and also we come up with a pair of optimal strategies for both the players.
What are the prices of random variables? In this paper, we define the least-squares prices of coin-flipping games, which are proved to be minimal, positive linear, and arbitrage-free. These prices depend both on a set of games that are available for investing simultaneously and on a risk-free interest rate. In addition, we show a case where the mean-variance portfolio theory is inappropriate.
We construct an approximate public-signal correlated equilibrium for a nonzero-sum differential game in the class of stochastic strategies with memory. The construction is based on a solution of an auxiliary nonzero-sum continuous-time stochastic game. This class of games includes stochastic differential games and continuous-time Markov games. Moreover, we study the limit of approximate equilibrium outcomes in the case when the auxiliary stochastic games tend to the original deterministic one. We show that it lies in the convex hull of the set of equilibrium values provided by deterministic pu…
In this paper we establish a new connection between a class of 2-player nonzero-sum games of optimal stopping and certain $2$-player nonzero-sum games of singular control. We show that whenever a Nash equilibrium in the game of stopping is attained by hitting times at two separate boundaries, then such boundaries also trigger a Nash equilibrium in the game of singular control. Moreover a differential link between the players' value functions holds across the two games.
In this article we consider zero and non-zero sum risk-sensitive average criterion games for semi-Markov processes with a finite state space. For the zero-sum case, under suitable assumptions we show that the game has a value. We also establish the existence of a stationary saddle point equilibrium. For the non-zero sum case, under suitable assumptions we establish the existence of a stationary Nash equilibrium.
This paper presents a theoretical framework for probably approximately correct (PAC) multi-agent reinforcement learning (MARL) algorithms for Markov games. The paper offers an extension to the well-known Nash Q-learning algorithm, using the idea of delayed Q-learning, in order to build a new PAC MARL algorithm for general-sum Markov games. In addition to guiding the design of a provably PAC MARL algorithm, the framework enables checking whether an arbitrary MARL algorithm is PAC. Comparative numerical results demonstrate performance and robustness.
Motivated by Generative Adversarial Networks, we study the computation of Nash equilibrium in concave network zero-sum games (NZSGs), a multiplayer generalization of two-player zero-sum games first proposed with linear payoffs. Extending previous results, we show that various game theoretic properties of convex-concave two-player zero-sum games are preserved in this generalization. We then generalize last iterate convergence results obtained previously in two-player zero-sum games. We analyze convergence rates when players update their strategies using Gradient Ascent, and its variant, Optimis…
We consider a finite state, finite action, zero-sum stochastic games with data defining the game lying in the ordered field of algebraic numbers. In both the discounted and the limiting average versions of these games we prove that the value vector also lies in the same field of algebraic numbers. In a prescribed sense, our results settle a problem that has remained open since, at least, 1991.
In the paper we consider the controlled continuous-time Markov chain describing the interacting particles system with the finite number of types. The system is controlled by two players with the opposite purposes. The limiting game as the number of particles tends to infinity is a zero-sum differential game. Krasovskii--Subbotin extremal shift provides the optimal strategy in the limiting game. The main result of the paper is the near optimality of the Krasovskii--Subbotin extremal shift rule for the original Markov game.
We consider two person zero-sum games where the players control, at discrete times {tn} induced by a partition $Π$ of R + , a continuous time Markov state process. We prove that the limit of the values v$Π$ exist as the mesh of $Π$ goes to 0. The analysis covers the cases of : 1) stochastic games (where both players know the state) 2) symmetric no information. The proof is by reduction to a deterministic differential game.
We study two classes of zero-sum stochastic games with compact action sets and a finite product state space. These two classes assume a communication property on the state spaces of the players. For strongly communicating on one side games, we prove the existence of the uniform value. For weakly communicating on both sides games, we prove that the asymptotic value, and therefore the uniform value, may fail to exist.
This paper proves several Tauberian theorems for general iterations of operators, and provides two applications to zero-sum stochastic games where the total payoff is a weighted sum of the stage payoffs. The first application is to provide conditions under which the existence of the asymptotic value implies the convergence of the values of the weighted game, as players get more and more patient. The second application concerns stochastic games with finite state space and action sets. This paper builds a simple class of asymptotically optimal strategies in the weighted game, that at each stage …
This paper considers a two-person zero-sum continuous-time Markov pure jump game in Borel state and action spaces over a fixed finite horizon. The main assumption on the model is the existence of a drift function, which bounds the reward rate. Under some regularity conditions, we show that the game has a value, and both of the players have their optimal policies.
We consider zero sum stochastic games. For every discount factor $λ$, a time normalization allows to represent the game as being played on the interval [0, 1]. We introduce the trajectories of cumulated expected payoff and of cumulated occupation measure up to time t $\in$ [0, 1], under $ε$-optimal strategies. A limit optimal trajectory is defined as an accumulation point as the discount factor tends to 0. We study existence, uniqueness and characterization of these limit optimal trajectories for absorbing games.
We study infinite horizon discounted-cost and ergodic-cost risk-sensitive zero-sum stochastic games for controlled continuous time Markov chains on a countable state space. For the discounted-cost game we prove the existence of value and saddle-point equilibrium in the class of Markov strategies under nominal conditions. For the ergodic-cost game we prove the existence of values and saddle point equilibrium by studying the corresponding Hamilton-Jacobi-Isaacs equation under a certain Lyapunov condition.
We analyze the convergence properties of the two-timescale fictitious play combining the classical fictitious play with the Q-learning for two-player zero-sum stochastic games with player-dependent learning rates. We show its almost sure convergence under the standard assumptions in two-timescale stochastic approximation methods when the discount factor is less than the product of the ratios of player-dependent step sizes. To this end, we formulate a novel Lyapunov function formulation and present a one-sided asynchronous convergence result.
In this paper we introduce the discontinuous universal feedback for the problem of Nash equilibrium in two person non-zero sum differential game. We assume that there exist functions satisfying some conditions analogous to the infinitesimal conditions on value function in zero sum differential games. Under this assumption we prove the existence of universal feedback Nash equilibrium.
We study two person nonzero-sum stochastic differential games with risk-sensitive discounted and ergodic cost criteria. Under certain conditions we establish a Nash equilibrium in Markov strategies for the discounted cost criterion and a Nash equilibrium in stationary strategies for the ergodic cost criterion. We achieve our results by studying the relevant systems of coupled HJB equations.
In the nonzero-sum setting, we establish a connection between Nash equilibria in games of optimal stopping (Dynkin games) and generalised Nash equilibrium problems (GNEP). In the Dynkin game this reveals novel equilibria of threshold type and of more complex types, and leads to novel uniqueness and stability results.
The value of a zero-sum differential games is known to exist, under Isaacs' condition, as the unique viscosity solution of a Hamilton-Jacobi-Bellman equation. In this note we provide a self-contained proof based on the construction of $\ep$-optimal strategies, which is inspired by the "extremal aiming" method from Krasovskii and Subbotin.
In this paper, we study a nonzero-sum stochastic differential game in Markovian framework. We show the existence of the Nash equilibrium point which is discontinuous and of bang-bang type under natural conditions. The main tool is the notion of backward stochastic differential equations which, in our case, are multidimensional with discontinuous generators with respect to z component.
This paper extends Berge's maximum theorem for possibly noncompact action sets and unbounded cost functions to minimax problems and studies applications of these extensions to two-player zero-sum games with possibly noncompact action sets and unbounded payoffs. For games with perfect information, also known under the name of turn-based games, this paper establishes continuity properties of value functions and solution multifunctions. For games with simultaneous moves, it provides results on the existence of lopsided values (the values in the asymmetric form) and solutions. This paper also esta…
We introduce a zero-sum game problem of mean-field type as an extension of the classical zero-sum Dynkin game problem to the case where the payoff processes might depend on the value of the game and its probability law. We establish sufficient conditions under which such a game admits a value and a saddle point. Furthermore, we provide a characterization of the value of the game in terms of a specific class of doubly reflected backward stochastic differential equations (BSDEs) of mean-field type, for which we derive an existence and uniqueness result. We then introduce a corresponding system o…
This article is related to risk-sensitive nonzero-sum stochastic differential games in the Markovian framework. This game takes into account the attitudes of the players toward risk and the utility is of exponential form. We show the existence of a Nash equilibrium point for the game when the drift is no longer bounded and only satisfies a linear growth condition. The main tool is the notion of backward stochastic differential equation, which in our case, is multidimensional with continuous generator involving both a quadratic term of Z and a stochastic linear growth component with respect to …
The infinite horizon risk-sensitive discounted-cost and ergodic-cost nonzero-sum stochastic games for controlled Markov chains with countably many states are analyzed. For the discounted-cost game, we prove the existence of Nash equilibrium strategies in the class of Markov strategies under fairly general conditions. Under an additional geometric ergodicity condition and a small cost criterion, the existence of Nash equilibrium strategies in the class of stationary Markov strategies is proved for the ergodic-cost game.
We prove that for a class of zero-sum differential games with incomplete information on both sides, the value admits a probabilistic representation as the value of a zero-sum stochastic differential game with complete information, where both players control a continuous martingale. A similar representation as a control problem over discontinuous martingales was known for games with incomplete information on one side (see Cardaliaguet-Rainer [8]), and our result is a continuous-time analog of the so called splitting-game introduced in Laraki [20] and Sorin [27] in order to analyze discrete-time…
We prove a Tauberian theorem for nonexpansive operators, and apply it to the model of zero-sum stochastic game. Under mild assumptions, we prove that the value of the lambda-discounted game v_{lambda} converges uniformly when lambda goes to 0 if and only if the value of the n-stage game v_n converges uniformly when n goes to infinity. This generalizes the Tauberian theorem of Lehrer and Sorin (1992) to the two-player zero-sum case. We also provide the first example of a stochastic game with public signals on the state and perfect observation of actions, with finite state space, signal sets and…
This paper provides sufficient conditions for the existence of solutions for two-person zero-sum games with inf/sup-compact payoff functions and with possibly noncompact decision sets for both players. Payoff functions may be unbounded, and we do not assume any convexity/concavity-type conditions. For such games expected payoff may not exist for some pairs of strategies. The results of this paper imply several classic facts. The paper also provides sufficient conditions for the existence of a value and solutions for each player. The results of this paper are illustrated with the number guessin…
In this paper, we introduce a homotopy function to trace the trajectory by applying modified homotopy continuation method for finding the solution of two-person zero-sum discounted stochastic ARAT game. We show that the algorithm has the higher order of convergence. For the proposed algorithm, the homotopy path approaching the solution is smooth and bounded. Two numerical examples are illustrated to show the effectiveness of the proposed algorithm.
The paper is concerned with a zero-sum Stackelberg stochastic linear-quadratic (LQ, for short) differential game over finite horizons. Under a fairly weak condition, the Stackelberg equilibrium is explicitly obtained by first solving a forward stochastic LQ optimal control problem (SLQ problem, for short) and then a backward SLQ problem. Two Riccati equations are derived in constructing the Stackelberg equilibrium. An interesting finding is that the difference of these two Riccati equations coincides with the Riccati equation associated with the zero-sum Nash stochastic LQ differential game, w…
Many emerging problems involve teams of agents taking part in a game. Such problems require a stochastic analysis with regard to the correlation structures among the agents belonging to a given team. In the context of Standard Borel spaces, this paper makes the following contributions for two teams of finitely many agents taking part in a zero-sum game: (i) An existence result will be presented for saddle-point equilibria in zero-sum games involving teams against teams when common randomness is assumed to be available in each team with an analysis on conditions for compactness of strategic tea…
Definable zero-sum stochastic games involve a finite number of states and action sets, reward and transition functions that are definable in an o-minimal structure. Prominent examples of such games are finite, semi-algebraic or globally subanalytic stochastic games. We prove that the Shapley operator of any definable stochastic game with separable transition and reward functions is definable in the same structure. Definability in the same structure does not hold systematically: we provide a counterexample of a stochastic game with semi-algebraic data yielding a non semi-algebraic but globally …
Multi-agent reinforcement learning (MARL) is often modeled using the framework of Markov games (also called stochastic games or dynamic games). Most of the existing literature on MARL concentrates on zero-sum Markov games but is not applicable to general-sum Markov games. It is known that the best-response dynamics in general-sum Markov games are not a contraction. Therefore, different equilibria in general-sum Markov games can have different values. Moreover, the Q-function is not sufficient to completely characterize the equilibrium. Given these challenges, model based learning is an attract…
Tsinghua University（清华大学）；Meta AI Research（Meta人工智能研究院）；Princeton University（普林斯顿大学）；University of Washington（华盛顿大学）
Policy-based methods with function approximation are widely used for solving two-player zero-sum games with large state and/or action spaces. However, it remains elusive how to obtain optimization and statistical guarantees for such algorithms. We present a new policy optimization algorithm with function approximation and prove that under standard regularity conditions on the Markov game and the function approximation class, our algorithm finds a near-optimal policy within a polynomial number of samples and iterations. To our knowledge, this is the first provably efficient policy optimization …
In this paper we study continuous-time two-player zero-sum optimal switching games on a finite horizon. Using the theory of doubly reflected BSDEs with interconnected barriers, we show that this game has a value and an equilibrium in the players' switching controls.
This paper is concerned with a non-zero sum differential game problem of an anticipated forward-backward stochastic differential delayed equation under partial information. We establish a necessary maximum principle and sufficient verification theorem of the game system by virtue of the duality and convex variational method. We apply the theoretical results and stochastic filtering theory to study a linear-quadratic game system and derive the explicit form of the Nash equilibrium point and discuss the existence and uniqueness in particular cases. As an application, we consider a time-delayed p…
The Stackelberg prediction game (SPG) is popular in characterizing strategic interactions between a learner and an attacker. As an important special case, the SPG with least squares loss (SPG-LS) has recently received much research attention. Although initially formulated as a difficult bi-level optimization problem, SPG-LS admits tractable reformulations which can be polynomially globally solved by semidefinite programming or second order cone programming. However, all the available approaches are not well-suited for handling large-scale datasets, especially those with huge numbers of feature…
In a zero-sum stochastic game, at each stage, two adversary players take decisions and receive a stage payoff determined by them and by a controlled random variable representing the state of nature. The total payoff is the normalized discounted sum of the stage payoffs. In this paper we solve the "constant payoff" conjecture formulated by Sorin, Vigeral and Venel (2010): if both players use optimal strategies, then for any alpha>0, the expected discounted payoff between stage 1 and stage alpha/lambda tends to the limit discounted value of the game, as the discount rate lambda goes to 0.
In this paper, an open-loop two-person non-zero sum stochastic differential game is considered for forward-backward stochastic systems. More precisely, the controlled systems are described by a fully coupled nonlinear multi- dimensional forward-backward stochastic differential equation driven by a multi-dimensional Brownian motion. one sufficient (a verification theorem) and one necessary conditions for the existence of open-loop Nash equilibrium points for the corresponding two-person non-zero sum stochastic differential game are proved. The control domain need to be convex and the admissible…
We study nonzero-sum stochastic differential games with risk-sensitive ergodic cost criterion. Under certain conditions, using multi-parameter eigenvalue approach, we establish the existence of a Nash equilibrium in the space of stationary Markov strategies. We achieve our results by studying the relevant systems of coupled HJB equations. Exploiting the stochastic representation of the principal eigenfunctions we completely characterize Nash equilibrium points in the space of stationary Markov strategies.
In this paper we study the zero-sum and nonzero-sum differential games with not assuming Isaacs condition. Along with the partition $π$ of the time interval $[0,T]$, we choose the suitable random non-anticipative strategy with delay to study our differential games with asymmetric information. Using Fenchel transformation, we prove that the limits of the upper value function $W^π$ and lower value function $V^π$ coincide when the mesh of partition $π$ tends to 0. Moreover, we give a characterization for the Nash equilibrium payoff (NEP, for short) of our nonzero-sum differential games without Is…
On a filtered probability space $(Ω,\mathcal{F}, (\mathcal{F}_t)_{t\in[0,\infty]}, \mathbb{P})$, we consider the two-player non-zero-sum stopping game $u^i := \mathbb{E}[U^i(ρ,τ)],\ i=1,2$, where the first player choose a stopping strategy $ρ$ to maximize $u^1$ and the second player chose a stopping strategy $τ$ to maximize $u^2$. Unlike the Dynkin game, here we assume that $U(s,t)$ is $\mathcal{F}_{s\vee t}$-measurable. Assuming the continuity of $U^i$ in $(s,t)$, we show that there exists an $ε$-Nash equilibrium for any $ε>0$.
This paper is concerned with a linear quadratic stochastic two-person zero-sum differential game with constant coefficients in an infinite time horizon. Open-loop and closed-loop saddle points are introduced. The existence of closed-loop saddle points is characterized by the solvability of an algebraic Riccati equation with a certain stabilizing condition. A crucial result makes our approach work is the unique solvability of a class of linear backward stochastic differential equations in an infinite horizon.
