A seminal result in game theory is von Neumann's minmax theorem, which states that zero-sum games admit an essentially unique equilibrium solution. Classical learning results build on this theorem to show that online no-regret dynamics converge to an equilibrium in a time-average sense in zero-sum games. In the past several years, a key research direction has focused on characterizing the day-to-day behavior of such dynamics. General results in this direction show that broad classes of online learning dynamics are cyclic, and formally Poincaré recurrent, in zero-sum games. We analyze the robustness of these online learning behaviors in the case of periodic zero-sum games with a time-invariant equilibrium. This model generalizes the usual repeated game formulation while also being a realistic and natural model of a repeated competition between players that depends on exogenous environmental variations such as time-of-day effects, week-to-week trends, and seasonality. Interestingly, time-average convergence may fail even in the simplest such settings, in spite of the equilibrium being fixed. In contrast, using novel analysis methods, we show that Poincaré recurrence provably generalizes despite the complex, non-autonomous nature of these dynamical systems.
Algorithmic game theory (AGT) focuses on the design and analysis of algorithms for interacting agents, with interactions rigorously formalized within the framework of games. Results from AGT find applications in domains such as online bidding auctions for web advertisements and network routing protocols. Monotone games are games where agent strategies naturally converge to an equilibrium state. Previous results in AGT have been obtained for convex, socially-convex, or smooth games, but not monotone games. Our primary theoretical contributions are defining the monotone game setting and its exte…
Chalmers University of Technology（查尔姆斯理工大学）；Microsoft Research Lab（微软研究院）
We study two-player general sum repeated finite games where the rewards of each player are generated from an unknown distribution. Our aim is to find the egalitarian bargaining solution (EBS) for the repeated game, which can lead to much higher rewards than the maximin value of both players. Our most important contribution is the derivation of an algorithm that achieves simultaneously, for both players, a high-probability regret bound of order $\mathcal{O}(\sqrt[3]{\ln T}\cdot T^{2/3})$ after any $T$ rounds of play. We demonstrate that our upper bound is nearly optimal by proving a lower bound…
PPA-Game: Characterizing and Learning Competitive Dynamics Among Online Content Creators
MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Institute for Theoretical Computer Science, Shanghai University of Finance and Economics（交叉学科计算与经济学联合实验室，理论计算机科学研究院，上海财经大学）；College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）；Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）；School of Economics and Management, Tsinghua University（经济管理学院，清华大学）
Maastricht University（马斯特里赫特大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Criteo AI Lab（科韬人工智能实验室）
We examine the long-run behavior of multi-agent online learning in games that evolve over time. Specifically, we focus on a wide class of policies based on mirror descent, and we show that the induced sequence of play (a) converges to Nash equilibrium in time-varying games that stabilize in the long run to a strictly monotone limit; and (b) it stays asymptotically close to the evolving equilibrium of the sequence of stage games (assuming they are strongly monotone). Our results apply to both gradient-based and payoff-based feedback - i.e., the "bandit feedback" case where players only get to o…
Online game playing algorithms produce high-quality strategies with a fraction of memory and computation required by their offline alternatives. Continual Resolving (CR) is a recent theoretically sound approach to online game playing that has been used to outperform human professionals in poker. However, parts of the algorithm were specific to poker, which enjoys many properties not shared by other imperfect information games. We present a domain-independent formulation of CR applicable to any two-player zero-sum extensive-form games that works with an abstract resolving algorithm. We further …
$Q$ Can Play That Game: Online Fitted $Q$-Iteration for Continuous-Action Zero-Sum Markov Games with Convex-Concave Function Approximation
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of California, Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology (MIT) Lincoln Labs（麻省理工学院林肯实验室）
The current state of the art in playing many important perfect information games, including Chess and Go, combines planning and deep reinforcement learning with self-play. We extend this approach to imperfect information games and present ExIt-OOS, a novel approach to playing imperfect information games within the Expert Iteration framework and inspired by AlphaZero. We use Online Outcome Sampling, an online search algorithm for imperfect information games in place of MCTS. While training online, our neural strategy is used to improve the accuracy of playouts in OOS, allowing a learning and pl…
Department of IEOR, Columbia University（哥伦比亚大学工业工程与运营研究系）；Mathematical Institute, University of Oxford（牛津大学数学系）；Citadel Securities（Citadel证券公司）
Reinforcement learning for multi-agent games has attracted lots of attention recently. However, given the challenge of solving Nash equilibria for large population games, existing works with guaranteed polynomial complexities either focus on variants of zero-sum and potential games, or aim at solving (coarse) correlated equilibria, or require access to simulators, or rely on certain assumptions that are hard to verify. This work proposes MF-OML (Mean-Field Occupation-Measure Learning), an online mean-field reinforcement learning algorithm for computing approximate Nash equilibria of large popu…
Search has played a fundamental role in computer game research since the very beginning. And while online search has been commonly used in perfect information games such as Chess and Go, online search methods for imperfect information games have only been introduced relatively recently. This paper addresses the question of what is a sound online algorithm in an imperfect information setting of two-player zero-sum games. We argue that the~fixed-strategy~definitions of exploitability and $ε$-Nash equilibria are ill-suited to measure an online algorithm's worst-case performance. We thus formalize…
High-Probability Nash Regret for Decentralized Learning in Markov $α$-Potential Games: Episodic and Fully Online Asynchronous Algorithms with Applications to Markov Congestion Games
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Università degli Studi di Milano（米兰大学）；Toulouse School of Economics（图卢兹经济学院）；IBM Research - Tokyo（IBM研究院（东京））；Luiss University（路易斯大学）；IBM Research - Haifa（IBM研究院（海法））
We study a repeated game between a supplier and a retailer who want to maximize their respective profits without full knowledge of the problem parameters. After characterizing the uniqueness of the Stackelberg equilibrium of the stage game with complete information, we show that even with partial knowledge of the joint distribution of demand and production costs, natural learning dynamics guarantee convergence of the joint strategy profile of supplier and retailer to the Stackelberg equilibrium of the stage game. We also prove finite-time bounds on the supplier's regret and asymptotic bounds o…
Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）；Department of Computing Science, University of Alberta（阿尔伯塔大学计算科学系）；Department of Computer Science, National Yang Ming Chiao Tung University（国立阳明交通大学资讯工程学系）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）
Myntra is an online fashion e-commerce company based in India. At Myntra, a market leader in fashion e-commerce in India, customer experience is paramount and a significant portion of our resources are dedicated to it. Here we describe an algorithm that identifies eligible customers to enable preferential product return processing for them by Myntra. We declare the group of aforementioned eligible customers on the platform as elite customers. Our algorithm to identify eligible/elite customers is based on sound principles of game theory. It is simple, easy to implement and scalable.
Competitive online games use rating systems to match players with similar skills to ensure a satisfying experience for players. In this paper, we focus on the importance of addressing different aspects of playing behavior when modeling players for creating match-ups. To this end, we engineer several behavioral features from a dataset of over 75,000 battle royale matches and create player models based on the retrieved features. We then use the created models to predict ranks for different groups of players in the data. The predicted ranks are compared to those of three popular rating systems. O…
We introduce a new solution concept, called periodicity, for selecting optimal strategies in strategic form games. This periodicity solution concept yields new insight into non-trivial games. In mixed strategy strategic form games, periodic solutions yield values for the utility function of each player that are equal to the Nash equilibrium ones. In contrast to the Nash strategies, here the payoffs of each player are robust against what the opponent plays. Sometimes, periodicity strategies yield higher utilities, and sometimes the Nash strategies do, but often the utilities of these two strate…
Motivated by the markets operating on fast time scales, we present a framework for online coalitional games with time-varying coalitional values and propose real-time payoff distribution mechanisms. Specifically, we design two online distributed algorithms to track the Shapley value and the core, the two most widely studied payoff distribution criteria in coalitional game theory. We show that the payoff distribution trajectory resulting from our proposed algorithms converges to a neighborhood of the time-varying solutions. We adopt an operator-theoretic perspective to show the convergence of o…
We study fast rates of convergence in the setting of nonparametric online regression, namely where regret is defined with respect to an arbitrary function class which has bounded complexity. Our contributions are two-fold: - In the realizable setting of nonparametric online regression with the absolute loss, we propose a randomized proper learning algorithm which gets a near-optimal cumulative loss in terms of the sequential fat-shattering dimension of the hypothesis class. In the setting of online classification with a class of Littlestone dimension $d$, our bound reduces to $d \cdot {\rm pol…
The Rock-Paper-Scissors (RPS) game is a classic non-cooperative game widely studied in terms of its theoretical analysis as well as in its applications, ranging from sociology and biology to economics. Many experimental results of the RPS game indicate that this game is better modelled by the discretized best-response dynamics rather than continuous time dynamics. In this work we show that the attractor of the discrete time best-response dynamics of the RPS game is finite and periodic. Moreover we also describe the bifurcations of the attractor and determine the exact number, period and locati…
KTH Royal Institute of Technology（KTH皇家理工学院）；Duke University（杜克大学）
We consider an online stochastic game with risk-averse agents whose goal is to learn optimal decisions that minimize the risk of incurring significantly high costs. Specifically, we use the Conditional Value at Risk (CVaR) as a risk measure that the agents can estimate using bandit feedback in the form of the cost values of only their selected actions. Since the distributions of the cost functions depend on the actions of all agents that are generally unobservable, they are themselves unknown and, therefore, the CVaR values of the costs are difficult to compute. To address this challenge, we p…
We present a novel control-theoretic understanding of online optimization and learning in games, via the notion of passivity. Passivity is a fundamental concept in control theory, which abstracts energy conservation and dissipation in physical systems. It has become a standard tool in analysis of general feedback systems, to which game dynamics belong. Our starting point is to show that all continuous-time Follow-the-Regularized-Leader (FTRL) dynamics, which include the well-known Replicator Dynamic, are lossless, i.e. it is passive with no energy dissipation. Interestingly, we prove that pass…
Optimistic Online Learning in Symmetric Cone Games
Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（新加坡国立大学）；Centre for Quantum Technologies（量子技术中心）；Archimedes/Athena Research Center（Archimedes/Athena研究中心）
Large-scale screening for potential threats with limited resources and capacity for screening is a problem of interest at airports, seaports, and other ports of entry. Adversaries can observe screening procedures and arrive at a time when there will be gaps in screening due to limited resource capacities. To capture this game between ports and adversaries, this problem has been previously represented as a Stackelberg game, referred to as a Threat Screening Game (TSG). Given the significant complexity associated with solving TSGs and uncertainty in arrivals of customers, existing work has assum…
We establish that algorithmic experiments in zero-sum games "fail miserably" to confirm the unique, sharp prediction of maxmin equilibration. Contradicting nearly a century of economic thought that treats zero-sum games nearly axiomatically as the exemplar symbol of economic stability, we prove that no meaningful prediction can be made about the day-to-day behavior of online learning dynamics in zero-sum games. Concretely, Multiplicative Weights Updates (MWU) with constant step-size is Lyapunov chaotic in the dual (payoff) space. Simply put, let's assume that an observer asks the agents playin…
We consider a class of concave continuous games in which the corresponding admissible strategy profile of each player underlies affine coupling constraints. We propose a novel algorithm that leads the relevant population dynamic toward Nash equilibrium. This algorithm is based on a mirror ascent algorithm, which suits with the framework of no-regret online learning, and on the augmented Lagrangian method. The decentralization aspect of the algorithm corresponds to the aspects that the iterate of each player requires the local information about how she contributes to the coupling constraints an…
Motivated by applications to online advertising and recommender systems, we consider a game-theoretic model with delayed rewards and asynchronous, payoff-based feedback. In contrast to previous work on delayed multi-armed bandits, we focus on multi-player games with continuous action spaces, and we examine the long-run behavior of strategic agents that follow a no-regret learning policy (but are otherwise oblivious to the game being played, the objectives of their opponents, etc.). To account for the lack of a consistent stream of information (for instance, rewards can arrive out of order, wit…
In this paper we introduce novel algorithmic strategies for effciently playing two-player games in which the players have different or identical player roles. In the case of identical roles, the players compete for the same objective (that of winning the game). The case with different player roles assumes that one of the players asks questions in order to identify a secret pattern and the other one answers them. The purpose of the first player is to ask as few questions as possible (or that the questions and their number satisfy some previously known constraints) and the purpose of the secret …
Access to online contents represents a large share of the Internet traffic. Most such contents are multimedia items which are user-generated, i.e., posted online by the contents' owners. In this paper we focus on how those who provide contents can leverage online platforms in order to profit from their large base of potential viewers. Actually, platforms like Vimeo or YouTube provide tools to accelerate the dissemination of contents, i.e., recommendation lists and other re-ranking mechanisms. Hence, the popularity of a content can be increased by paying a cost for advertisement: doing so, it w…
Online Parameter Identification of Generalized Non-cooperative Game
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Tongji University（同济大学）；Shanghai Research Institute for Intelligent Autonomous Systems（上海自主智能无人系统科学中心）
Existing game-theoretic planning methods assume that the robot knows the objective functions of the other agents a priori while, in practical scenarios, this is rarely the case. This paper introduces LUCIDGames, an inverse optimal control algorithm that is able to estimate the other agents' objective functions in real time, and incorporate those estimates online into a receding-horizon game-theoretic planner. LUCIDGames solves the inverse optimal control problem by recasting it in a recursive parameter-estimation framework. LUCIDGames uses an unscented Kalman filter (UKF) to iteratively update…
At a mixed Nash equilibrium, the payoff of a player does not depend on her own action, as long as her opponent sticks to his. In a periodic strategy, a concept developed in a previous paper (arXiv:1307.2035v4), in contrast, the own payoff does not depend on the opponent's action. Here, we generalize this to multi-player simultaneous perfect information strategic form games. We show that also in this class of games, there always exists at least one periodic strategy, and we investigate the mathematical properties of such periodic strategies. In addition, we demonstrate that periodic strategies …
Consider an important meeting to be held in a team-based organization. Taking availability constraints into account, an online scheduling poll is being used in order to decide upon the exact time of the meeting. Decisions are to be taken during the meeting, therefore each team would like to maximize its relative attendance in the meeting (i.e., the proportional number of its participating team members). We introduce a corresponding game, where each team can declare (in the scheduling poll) a lower total availability, in order to improve its relative attendance---the pay-off. We are especially …
We investigate the performance of concurrent remote sensing from independent strategic sources, whose goal is to minimize a linear combination of the freshness of information and the updating cost. In the literature, this is often investigated from a static perspective of setting the update rate of the sources a priori, either in a centralized optimal way or with a distributed game-theoretic approach. However, we argue that truly rational sources would better make such a decision with full awareness of the current age of information, resulting in a more efficient implementation of the updating…
