arXiv 2402.095492024-02-14cs.GT

博弈学习中帕累托最优的算法

Pareto-Optimal Algorithms for Learning in Games

Eshwar Ram Arunachaleswaran, Natalie Collina, Jon Schneider

AI总结：

研究重复博弈中的最优学习算法，提出渐近帕累托最优概念，证明无交换遗憾算法是帕累托最优的充分条件，并引入渐近菜单证明此类算法策略等价。

摘要：

我们研究如何刻画最优学习算法的问题，该算法用于在收益未知的对手面前进行重复博弈。在此问题中，第一玩家（称为学习者）承诺采用一种学习算法对抗第二玩家（称为优化器），而优化器通过为其（未知但明确的）收益选择最优动态策略来进行最佳响应。经典学习算法（如无遗憾算法）为学习者提供了一些反事实保证，但在面对特定优化器收益时，其表现可能比其他学习算法差得多。在本文中，我们引入了渐近帕累托最优学习算法的概念。直观地说，如果一种学习算法是帕累托最优的，则不存在另一种算法，能在面对所有优化器时渐近表现至少同样好，且在面对某些优化器时表现严格更好（至少好$Ω(T)$）。我们证明了诸如Multiplicative Weights和Follow The Regularized Leader等著名的无遗憾算法是帕累托受支配的。然而，尽管无遗憾不足以保证帕累托最优性，我们证明了严格更强的属性——无交换遗憾，是帕累托最优的充分条件。证明这些结果需要我们解决重复博弈特有的各种技术挑战，包括缺乏对长期理性的优化器如何在多轮博弈中对学习算法进行最佳响应的简单刻画。为此，我们引入了学习算法的渐近菜单概念：对手可渐近实现的策略轮廓上所有相关分布的凸闭包。我们证明所有无交换遗憾算法共享相同的渐近菜单，这意味着所有无交换遗憾算法是“策略等价的”。

英文摘要：

We study the problem of characterizing optimal learning algorithms for playing repeated games against an adversary with unknown payoffs. In this problem, the first player (called the learner) commits to a learning algorithm against a second player (called the optimizer), and the optimizer best-responds by choosing the optimal dynamic strategy for their (unknown but well-defined) payoff. Classic learning algorithms (such as no-regret algorithms) provide some counterfactual guarantees for the learner, but might perform much more poorly than other learning algorithms against particular optimizer payoffs. In this paper, we introduce the notion of asymptotically Pareto-optimal learning algorithms. Intuitively, if a learning algorithm is Pareto-optimal, then there is no other algorithm which performs asymptotically at least as well against all optimizers and performs strictly better (by at least $Ω(T)$) against some optimizer. We show that well-known no-regret algorithms such as Multiplicative Weights and Follow The Regularized Leader are Pareto-dominated. However, while no-regret is not enough to ensure Pareto-optimality, we show that a strictly stronger property, no-swap-regret, is a sufficient condition for Pareto-optimality. Proving these results requires us to address various technical challenges specific to repeated play, including the fact that there is no simple characterization of how optimizers who are rational in the long-term best-respond against a learning algorithm over multiple rounds of play. To address this, we introduce the idea of the asymptotic menu of a learning algorithm: the convex closure of all correlated distributions over strategy profiles that are asymptotically implementable by an adversary. We show that all no-swap-regret algorithms share the same asymptotic menu, implying that all no-swap-regret algorithms are ``strategically equivalent''.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文