We argue that the existing regret matchings for Nash equilibrium approximation conduct "jumpy" strategy updating when the probabilities of future plays are set to be proportional to positive regret measures. We propose a geometrical regret matching which features "smooth" strategy updating. Our approach is simple, intuitive and natural. The analytical and numerical results show that, continuously and "smoothly" suppressing "unprofitable" pure strategies is sufficient for the game to evolve towards Nash equilibrium, suggesting that in reality the tendency for equilibrium could be pervasive and irresistible. Technically, iterative regret matching gives rise to a sequence of adjusted mixed strategies for our study its approximation to the true equilibrium point. The sequence can be studied in metric space and visualized nicely as a clear path towards an equilibrium point. Our theory has limitations in optimizing the approximation accuracy.
A dominant approach to solving large imperfect-information games is Counterfactural Regret Minimization (CFR). In CFR, many regret minimization problems are combined to solve the game. For very large games, abstraction is typically needed to render CFR tractable. Abstractions are often manually tuned, possibly removing important strategic differences in the full game and harming performance. Function approximation provides a natural solution to finding good abstractions to approximate the full game. A common approach to incorporating function approximation is to learn the inputs needed for a r…
The regret matching algorithm proposed by Sergiu Hart is one of the most powerful iterative methods in finding correlated equilibrium. However, it is possibly not efficient enough, especially in large scale problems. We first rewrite the algorithm in a computationally practical way based on the idea of the regret matrix. Moreover, the rewriting makes the original algorithm more easy to understand. Then by some modification to the original algorithm, we introduce a novel variant, namely faster regret matching. The experiment result shows that the novel algorithm has a speed advantage comparing …
Adaptive Bandit Algorithms for Contextual Matching Markets
Center for Statistical Science, School of Mathematical Sciences, Peking University（北京大学数学科学学院统计科学中心）；INRIA, FairPlay Joint Team（INRIA公平玩联合团队）；Criteo AI lab, FairPlay Joint Team（Criteo AI实验室，公平玩联合团队）；Technion - Israel Institute of Technology（技术学院-以色列理工学院）
Making an informed decision -- for example, when choosing a career or housing -- requires knowledge about the available options. Such knowledge is generally acquired through costly trial and error, but this learning process can be disrupted by competition. In this work, we study how competition affects the long-term outcomes of individuals as they learn. We build on a line of work that models this setting as a two-sided matching market with bandit learners. A recent result in this area states that it is impossible to simultaneously guarantee two natural desiderata: stability and low optimal re…
Blackwell approachability is a framework for reasoning about repeated games with vector-valued payoffs. We introduce predictive Blackwell approachability, where an estimate of the next payoff vector is given, and the decision maker tries to achieve better performance based on the accuracy of that estimator. In order to derive algorithms that achieve predictive Blackwell approachability, we start by showing a powerful connection between four well-known algorithms. Follow-the-regularized-leader (FTRL) and online mirror descent (OMD) are the most prevalent regret minimizers in online convex optim…
Northwestern University（西北大学）；Yale University（耶鲁大学）；University of California, Berkeley（加州大学伯克利分校）
We study a Markov matching market involving a planner and a set of strategic agents on the two sides of the market. At each step, the agents are presented with a dynamical context, where the contexts determine the utilities. The planner controls the transition of the contexts to maximize the cumulative social welfare, while the agents aim to find a myopic stable matching at each step. Such a setting captures a range of applications including ridesharing platforms. We formalize the problem by proposing a reinforcement learning framework that integrates optimistic value iteration with maximum we…
Convergence of Regret Matching in Potential Games and Constrained Optimization
Carnegie Mellon University（卡内基梅隆大学）；Foundations of Cooperative AI Lab (FOCAL)（合作人工智能基础实验室）；University of California, Irvine（加利福尼亚大学欧文分校）；Massachusetts Institute of Technology（麻省理工学院）；University of Oxford（牛津大学）；Strategy Robot, Inc.（Strategy Robot 公司）；Strategic Machine, Inc.（Strategic Machine 公司）；Optimized Markets, Inc.（Optimized Markets 公司）
Harvard University（哈佛大学）；UC Santa Cruz（加州大学圣克鲁兹分校）
We address the question of repeatedly learning linear classifiers against agents who are strategically trying to game the deployed classifiers, and we use the Stackelberg regret to measure the performance of our algorithms. First, we show that Stackelberg and external regret for the problem of strategic classification are strongly incompatible: i.e., there exist worst-case scenarios, where any sequence of actions providing sublinear external regret might result in linear Stackelberg regret and vice versa. Second, we present a strategy-aware algorithm for minimizing the Stackelberg regret for w…
In general, two-agent decision-making problems can be modeled as a two-player game, and a typical solution is to find a Nash equilibrium in such game. Counterfactual regret minimization (CFR) is a well-known method to find a Nash equilibrium strategy in a two-player zero-sum game with imperfect information. The CFR method adopts a regret matching algorithm iteratively to reduce regret values progressively, enabling the average strategy to approach a Nash equilibrium. Although CFR-based methods have achieved significant success in the field of imperfect information games, there is still scope f…
Function approximation is a powerful approach for structuring large decision problems that has facilitated great achievements in the areas of reinforcement learning and game playing. Regression counterfactual regret minimization (RCFR) is a simple algorithm for approximately solving imperfect information games with normalized rectified linear unit (ReLU) parameterized policies. In contrast, the more conventional softmax parameterization is standard in the field of reinforcement learning and yields a regret bound with a better dependence on the number of actions. We derive approximation error-a…
