Many recent practical and theoretical breakthroughs focus on adversarial team multi-player games (ATMGs) in ex ante correlation scenarios. In this setting, team members are allowed to coordinate their strategies only before the game starts. Although there existing algorithms for solving extensive-form ATMGs, the size of the game tree generated by the previous algorithms grows exponentially with the number of players. Therefore, how to deal with large-scale zero-sum extensive-form ATMGs problems close to the real world is still a significant challenge. In this paper, we propose a generic multi-player transformation algorithm, which can transform any multi-player game tree satisfying the definition of AMTGs into a 2-player game tree, such that finding a team-maxmin equilibrium with correlation (TMECor) in large-scale ATMGs can be transformed into solving NE in 2-player games. To achieve this goal, we first introduce a new structure named private information pre-branch, which consists of a temporary chance node and coordinator nodes and aims to make decisions for all potential private information on behalf of the team members. We also show theoretically that NE in the transformed 2-player game is equivalent TMECor in the original multi-player game. This work significantly reduces the growth of action space and nodes from exponential to constant level. This enables our work to outperform all the previous state-of-the-art algorithms in finding a TMECor, with 182.89, 168.47, 694.44, and 233.98 significant improvements in the different Kuhn Poker and Leduc Poker cases (21K3, 21K4, 21K6 and 21L33). In addition, this work first practically solves the ATMGs in a 5-player case which cannot be conducted by existing algorithms.
Tongji University(同济大学);Shanghai Research Institute for Intelligent Autonomous Systems(上海智能自主系统研究院);Institute for Advanced Study, Tongji University(同济大学高等研究院)
Efficiently Computing Nash Equilibria in Adversarial Team Markov Games
University of California, Irvine(加州大学欧文分校);Carnegie Mellon University(卡内基梅隆大学);Columbia University(哥伦比亚大学);University of California, Santa Cruz(加州大学圣克鲁兹分校)
纳什均衡策略的求解是多智能体强化学习的核心问题,在理论和实践中都受到了广泛的关注。然而,迄今为止,可证明的保证要么仅限于完全竞争或合作的情况,要么强加了在大多数实际应用中难以满足的强有力的假设。在这项工作中,我们离开这些先前的结果调查无限地平线\emph{adversarial team Markov games},一个自然的和动机良好的一类游戏中,一队同样感兴趣的球员-在没有任何明确的协调或沟通-是竞争对手的球员。这种设置允许零和马尔可夫博弈和马尔可夫势博弈的统一处理,并作为一个步骤,以模拟更现实的战略互动,同时具有竞争和合作的利益。我们的主要贡献是第一个算法计算平稳$ε$ -近似纳什均衡的对抗团队马尔可夫游戏的计算复杂性是多项式的所有自然参数的游戏,以及$1/ε$。所提出的算法是特别自然和实用的,它是基于执行独立的策略梯度步骤,为每个球员在团队中,在串联与最佳对策从对手的一方,反过来,对手的政策,然后通过求解一个精心构造的线性规划。我们的分析利用非标准的技术来建立KKT最优性条件的非凸约束的非线性规划,从而导致自然的解释诱导拉格朗日乘子。一路上,我们显着扩展了对抗(正常形式)团队游戏中的最优策略的重要特征,由于Von Stengel和Koller(GEB '97)。