In a single-state repeated game, zero-determinant strategies can unilaterally force functions of the payoffs to take values in particular closed intervals. When the explicit use of a determinant is absent from the analysis, they are instead called autocratic. We extend their study to the setting of finite state games with deterministic transitions. For a given game we find that the endpoints of the intervals of enforceable values must satisfy fixed point equations. From these extreme enforceable values we show it is always possible to construct finite memory strategies to enforce a particular value. An algorithm is presented which will approximate the enforceable values in each state. Finally, we present formulas from which the exact solutions can be recovered from the approximate solution.
This paper studies the mechanisms, implications, and potential applications of the recently discovered class of Zero Determinant (ZD) strategies in iterated 2x2 games. These strategies were reported to successfully extort pure economic maximizers, and to mischievously determine the set of feasible long-term payoffs in iterated Prisoners' Dilemma by enforcing linear constraints on both players' expected average scores. These results are generalized for all symmetric 2x2 games and a general Battle of the Sexes, exemplified by four common games. Additionally, a comparison to conventional strategi…
University of Electronic Science and Technology of China（电子科技大学）；Shanghai Jiao Tong University（上海交通大学）
Repeated game has long been the touchstone model for agents' long-run relationships. Previous results suggest that it is particularly difficult for a repeated game player to exert an autocratic control on the payoffs since they are jointly determined by all participants. This work discovers that the scale of a player's capability to unilaterally influence the payoffs may have been much underestimated. Under the conventional iterated prisoner's dilemma, we develop a general framework for controlling the feasible region where the players' payoff pairs lie. A control strategy player is able to co…
We examine perfect information stochastic mean-payoff games - a class of games containing as special sub-classes the usual mean-payoff games and parity games. We show that deterministic memoryless strategies that are optimal for discounted games with state-dependent discount factors close to 1 are optimal for priority mean-payoff games establishing a strong link between these two classes.
We prove that optimal strategies exist in every perfect-information stochastic game with finitely many states and actions and a tail winning condition.
Repeated games have provided an explanation how mutual cooperation can be achieved even if defection is more favorable in a one-shot game in prisoner's dilemma situation. Recently found zero-determinant strategies have substantially been investigated in evolutionary game theory. The original memory-one zero-determinant strategies unilaterally enforce linear relations between average payoffs of players. Here, we extend the concept of zero-determinant strategies to memory-two strategies in repeated games. Memory-two zero-determinant strategies unilaterally enforce linear relations between correl…
Extensive-form games with imperfect recall are an important game-theoretic model that allows a compact representation of strategies in dynamic strategic interactions. Practical use of imperfect recall games is limited due to negative theoretical results: a Nash equilibrium does not have to exist, computing maxmin strategies is NP-hard, and they may require irrational numbers. We present the first algorithm for approximating maxmin strategies in two-player zero-sum imperfect recall games without absentmindedness. We modify the well-known sequence-form linear program to model strategies in imper…
The solution of parity games over pushdown graphs (Walukiewicz '96) was the first step towards an effective theory of infinite-state games. It was shown that winning strategies for pushdown games can be implemented again as pushdown automata. We continue this study and investigate the connection between game presentations and winning strategies in altogether six cases of game arenas, among them realtime pushdown systems, visibly pushdown systems, and counter systems. In four cases we show by a uniform proof method that we obtain strategies implementable by the same type of pushdown machine as …
Expectation-enforcing strategies for repeated games
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；NSF-Simons National Institute for Theory and Mathematics in Biology（美国国家科学基金会-西蒙斯生物理论与数学国家研究所）
Repeated games are difficult to analyze, especially when agents play mixed strategies. We study one-memory strategies in iterated prisoner's dilemma, then generalize the result to k-memory strategies in repeated games. Our result shows that there always exists a pure strategy best response, which can be computed with SMT or MDP solvers. However, there may not exist such pure strategy best response in multi-agent tournaments. All source code is released for verification.
Infinite-duration games with disturbances extend the classical framework of infinite-duration games, which captures the reactive synthesis problem, with a discrete measure of resilience against non-antagonistic external influence. This concerns events where the observed system behavior differs from the intended one prescribed by the controller. For games played on finite arenas it is known that computing optimally resilient strategies only incurs a polynomial overhead over solving classical games. This paper studies safety games with disturbances played on infinite arenas induced by pushdown s…
We consider perfect-information reachability stochastic games for 2 players on infinite graphs. We identify a subclass of such games, and prove two interesting properties of it: first, Player Max always has optimal strategies in games from this subclass, and second, these games are strongly determined. The subclass is defined by the property that the set of all values can only have one accumulation point -- 0. Our results nicely mirror recent results for finitely-branching games, where, on the contrary, Player Min always has optimal strategies. However, our proof methods are substantially diff…
We examine the problem of the existence of optimal deterministic stationary strategiesintwo-players antagonistic (zero-sum) perfect information stochastic games with finitely many states and actions.We show that the existenceof such strategies follows from the existence of optimal deterministic stationarystrategies for some derived one-player games.Thus we reducethe problem from two-player to one-player games (Markov decisionproblems), where usually it is much easier to tackle.The reduction is very general, it holds not only for all possible payoff mappings but alsoin more a general situations…
We describe an algorithm for computing best response strategies in a class of two-player infinite games of incomplete information, defined by payoffs piecewise linear in agents' types and actions, conditional on linear comparisons of agents' actions. We show that this class includes many well-known games including a variety of auctions and a novel allocation game. In some cases, the best-response algorithm can be iterated to compute Bayes-Nash equilibria. We demonstrate the efficiency of our approach on existing and new games.
Given a bimatrix game, the associated leadership or commitment games are defined as the games at which one player, the leader, commits to a (possibly mixed) strategy and the other player, the follower, chooses his strategy after having observed the irrevocable commitment of the leader. Based on a result by von Stengel and Zamir [2010], the notions of commitment value and commitment optimal strategies for each player are discussed as a possible solution concept. It is shown that in non-degenerate bimatrix games (a) pure commitment optimal strategies together with the follower's best response co…
This paper studies two-player zero-sum stochastic Bayesian games where each player has its own dynamic state that is unknown to the other player. Using typical techniques, we provide the recursive formulas and sufficient statistics in both the primal game and its dual games. It's also shown that with a specific initial parameter, the optimal strategy of one player in a dual game is also the optimal strategy of the player in the primal game. To deal with the long finite Bayesian game we have provided an algorithm to compute the sub-optimal strategies of the players step by step to avoid the LP …
We investigate concurrent two-player win/lose stochastic games on finite graphs with prefix-independent objectives. We characterize subgame optimal strategies and use this characterization to show various memory transfer results: 1) For a given (prefix-independent) objective, if every game that has a subgame almost-surely winning strategy also has a positional one, then every game that has a subgame optimal strategy also has a positional one; 2) Assume that the (prefix-independent) objective has a neutral color. If every turn-based game that has a subgame almost-surely winning strategy also ha…
This paper introduces algorithm instance games (AIGs) as a conceptual classification applying to games in which outcomes are resolved from joint strategies algorithmically. For such games, a fundamental question asks: How do the details of the algorithm's description influence agents' strategic behavior? We analyze two versions of an AIG based on the set-cover optimization problem. In these games, joint strategies correspond to instances of the set-cover problem, with each subset (of a given universe of elements) representing the strategy of a single agent. Outcomes are covers computed from th…
We prove that a deterministic n-person shortest path game has a Nash equlibrium in pure and stationary strategies if it is edge-symmetric (that is (u,v) is a move whenever (v,u) is, apart from moves entering terminal vertices) and the length of every move is positive for each player. Both conditions are essential, though it remains an open problem whether there exists a NE-free 2-person non-edge-symmetric game with positive lengths. We provide examples for NE-free 2-person edge-symmetric games that are not positive. We also consider the special case of terminal games (shortest path games in wh…
We extend the open games framework for compositional game theory to encompass also mixed strategies, making essential use of the discrete probability distribution monad. We show that the resulting games form a symmetric monoidal category, which can be used to compose probabilistic games in parallel and sequentially. We also consider morphisms between games, and show that intuitive constructions give rise to functors and adjunctions between pure and probabilistic open games.
Zero-determinant strategies are a class of memory-one strategies in repeated games which unilaterally enforce linear relationships between payoffs. It has long been unclear for what stage games zero-determinant strategies exist. We provide a necessary and sufficient condition for the existence of zero-determinant strategies. This condition can be interpreted as the existence of two different actions which unilaterally adjust the total value of a linear combination of payoffs. A relation between the class of stage games where zero-determinant strategies exist and other class of stage games is a…
We propose a new interpretation of the strange phenomena that some authors have observed about the Wald game. This interpretation is possible thanks to the new language of \emph{loadings} that Morrison and the author have introduced in a previous work. Using the theory of loadings and allowed strategies, we are also able to prove that Wald's game admits a \emph{natural} solution and, as one can expect, the game turns out to be fair for this solution. As a technical tool, we introduce the notion of \emph{embedding a game into another game} that could be of interest from a theoretical point of v…
We investigate determinacy of delay games with Borel winning conditions, infinite-duration two-player games in which one player may delay her moves to obtain a lookahead on her opponent's moves. First, we prove determinacy of such games with respect to a fixed evolution of the lookahead. However, strategies in such games may depend on information about the evolution. Thus, we introduce different notions of universal strategies for both players, which are evolution-independent, and determine the exact amount of information a universal strategy needs about the history of a play and the evolution…
We introduce a new solution concept, called periodicity, for selecting optimal strategies in strategic form games. This periodicity solution concept yields new insight into non-trivial games. In mixed strategy strategic form games, periodic solutions yield values for the utility function of each player that are equal to the Nash equilibrium ones. In contrast to the Nash strategies, here the payoffs of each player are robust against what the opponent plays. Sometimes, periodicity strategies yield higher utilities, and sometimes the Nash strategies do, but often the utilities of these two strate…
We prove the existence and computability of optimal strategies in weighted limit games, zero-sum infinite-duration games with a Büchi-style winning condition requiring to produce infinitely many play prefixes that satisfy a given regular specification. Quality of plays is measured in the maximal weight of infixes between successive play prefixes that satisfy the specification.
We consider infinite-state turn-based stochastic games of two players, Box and Diamond, who aim at maximizing and minimizing the expected total reward accumulated along a run, respectively. Since the total accumulated reward is unbounded, the determinacy of such games cannot be deduced directly from Martin's determinacy result for Blackwell games. Nevertheless, we show that these games are determined both for unrestricted (i.e., history-dependent and randomized) strategies and deterministic strategies, and the equilibrium value is the same. Further, we show that these games are generally not d…
We initiate a study of random instances of nonlocal games. We show that quantum strategies are better than classical for almost any 2-player XOR game. More precisely, for large n, the entangled value of a random 2-player XOR game with n questions to every player is at least 1.21... times the classical value, for 1-o(1) fraction of all 2-player XOR games.
We prove the existence and computability of optimal strategies in weighted limit games, zero-sum infinite-duration games with a Büchi-style winning condition requiring to produce infinitely many play prefixes that satisfy a given regular specification. Quality of plays is measured in the maximal weight of infixes between successive play prefixes that satisfy the specification.
We investigate symmetric equilibria of mutual reinforcement learning when both players alternately learn the optimal memory-two strategies against the opponent in the repeated prisoners' dilemma game. We provide a necessary condition for memory-two deterministic strategies to form symmetric equilibria. We then provide three examples of memory-two deterministic strategies which form symmetric mutual reinforcement learning equilibria. We also prove that mutual reinforcement learning equilibria formed by memory-two strategies are also mutual reinforcement learning equilibria when both players use…
We use replicator dynamics to study an iterated prisoners' dilemma game with memory. In this study, we investigate the characteristics of all 32 possible strategies with a single-step memory by observing the results when each strategy encounters another one. Based on these results, we define similarity measures between the 32 strategies and perform a network analysis of the relationship between the strategies by constructing a strategies network. Interestingly, we find that a win-lose circulation, like rock-paper-scissors, exists between strategies and that the circulation results from one unu…
Game theory is the mathematical framework for analyzing strategic interactions in conflict and competition situations. In recent years quantum game theory has earned the attention of physicists, and has emerged as a branch of quantum information theory [1]. With the aid of entanglement and linear superposition of strategies, quantum games are shown to yield signifcant advantage over their classical counterparts. In this paper we explore optimal and equilibrium solutions to quantum minority games. Initial states with different level of entanglement are investigated. Focus will be on 4 and 6 pla…
This paper studies the optimization of strategies in the context of possibly randomized two players zero-sum games with incomplete information. We compare 5 algorithms for tuning the parameters of strategies over a benchmark of 12 games. A first evolutionary approach consists in designing a highly randomized opponent (called naive opponent) and optimizing the parametric strategy against it; a second one is optimizing iteratively the strategy, i.e. constructing a sequence of strategies starting from the naive one. 2 versions of coevolutions, real and approximate, are also tested as well as a se…
Imitation is simple behavior which uses successful actions of others in order to deal with one's own problems. Because success of imitation generally depends on whether profit of an imitating agent coincides with those of other agents or not, game theory is suitable for specifying situations where imitation can be successful. One of the concepts describing successfulness of imitation in repeated two-player symmetric games is unbeatability. For infinitely repeated two-player symmetric games, a necessary and sufficient condition for some imitation strategy to be unbeatable was specified. However…
Two-player stochastic games are games with two 2 players and a randomised entity called "nature". A natural question to ask in this framework is the existence of strategies that ensure that an event happens with probability 1 (almost-sure strategies). In the case of Markov decision processes, when the event 2 of interest is given as a parity condition, we can replace the "nature" by two more players that play according to the rules of what is known as Banach-Mazur game [1]. In this paper we continue this research program by extending the above result to two-player stochastic parity games. As i…
We consider two-player partial-observation stochastic games on finite-state graphs where player 1 has partial observation and player 2 has perfect observation. The winning condition we study are ω-regular conditions specified as parity objectives. The qualitative-analysis problem given a partial-observation stochastic game and a parity objective asks whether there is a strategy to ensure that the objective is satisfied with probability~1 (resp. positive probability). These qualtitative-analysis problems are known to be undecidable. However in many applications the relevant question is the exis…
We develop a new approach that computes approximate equilibrium strategies in Jotto, a popular word game. Jotto is an extremely large two-player game of imperfect information; its game tree has many orders of magnitude more states than games previously studied, including no-limit Texas hold 'em. To address the fact that the game is so large, we propose a novel strategy representation called oracular form, in which we do not explicitly represent a strategy, but rather appeal to an oracle that quickly outputs a sample move from the strategy's distribution. Our overall approach is based on an ext…
We consider imperfect information stochastic games where we require the players to use pure (i.e. non randomised) strategies. We consider reachability, safety, Büchi and co-Büchi objectives, and investigate the existence of almost-sure/positively winning strategies for the first player when the second player is perfectly informed or more informed than the first player. We obtain decidability results for positive reachability and almost-sure Büchi with optimal algorithms to decide existence of a pure winning strategy and to compute one if exists. We complete the picture by showing that positive…
Zero-determinant (ZD) strategies, a recently found novel class of strategies in repeated games, has attracted much attention in evolutionary game theory. A ZD strategy unilaterally enforces a linear relation between average payoffs of players. Although existence and evolutional stability of ZD strategies have been studied in simple games, their mathematical properties have not been well-known yet. For example, what happens when more than one players employ ZD strategies have not been clarified. In this paper, we provide a general framework for investigating situations where more than one playe…
DeepMind（深度思维）；University College London（伦敦大学学院）；Google（谷歌）；Carnegie Mellon University（卡内基梅隆大学）
Rating strategies in a game is an important area of research in game theory and artificial intelligence, and can be applied to any real-world competitive or cooperative setting. Traditionally, only transitive dependencies between strategies have been used to rate strategies (e.g. Elo), however recent work has expanded ratings to utilize game theoretic solutions to better rate strategies in non-transitive games. This work generalizes these ideas and proposes novel algorithms suitable for N-player, general-sum rating of strategies in normal-form games according to the payoff rating system. This …
Two-player games on graphs is central in many problems in formal verification and program analysis such as synthesis and verification of open systems. In this work we consider solving recursive game graphs (or pushdown game graphs) that can model the control flow of sequential programs with recursion. While pushdown games have been studied before with qualitative objectives, such as reachability and $ω$-regular objectives, in this work we study for the first time such games with the most well-studied quantitative objective, namely, mean-payoff objectives. In pushdown games two types of strateg…
The Prisoners' Dilemma is perhaps the most famous model in the field of game theory. Consequently, it is natural to investigate its quantum version when one considers to apply quantum strategies to game theory. There are two main results in this paper: 1) The well-known Prisoners' Dilemma can be categorized into three types and only the third type is adaptable for quantum strategies. 2) As a reverse problem of game theory, mechanism design provides a better circumstance for quantum strategies than game theory does.
In game theory, an Evolutionarily Stable Set (ES set) is a set of Nash Equilibrium (NE) strategies that give the same payoffs. Similar to an Evolutionarily Stable Strategy (ES strategy), an ES set is also a strict NE. This work investigates the evolutionary stability of classical and quantum strategies in the quantum penny flip games. In particular, we developed an evolutionary game theory model to conduct a series of simulations where a population of mixed classical strategies from the ES set of the game were invaded by quantum strategies. We found that when only one of the two players' mixed…
Network congestion games are a convenient model for reasoning about routing problems in a network: agents have to move from a source to a target vertex while avoiding congestion, measured as a cost depending on the number of players using the same link. Network congestion games have been extensively studied over the last 40 years, while their extension with timing constraints were considered more recently. Most of the results on network congestion games consider blind strategies: they are static, and do not adapt to the strategies selected by the other players. We extend the recent results of …
What is a finite-state strategy in a delay game? We answer this surprisingly non-trivial question by presenting a very general framework that allows to remove delay: finite-state strategies exist for all winning conditions where the resulting delay-free game admits a finite-state strategy. The framework is applicable to games whose winning condition is recognized by an automaton with an acceptance condition that satisfies a certain aggregation property. Our framework also yields upper bounds on the complexity of determining the winner of such delay games and upper bounds on the necessary looka…
