Shanghai Jiao Tong University（上海交通大学）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；University College London（伦敦大学学院）；Shanghai Digital Brain Laboratory（上海数字大脑研究院）
Ming Zhou, Jingxiao Chen, Ying Wen, Weinan Zhang, Yaodong Yang, Yong Yu, Jun Wang
备注：
improved theoretical results; fixed some typos; 22 pages, 8 figures
Policy Space Response Oracle methods (PSRO) provide a general solution to learn Nash equilibrium in two-player zero-sum games but suffer from two drawbacks: (1) the computation inefficiency due to the need for consistent meta-game evaluation via simulations, and (2) the exploration inefficiency due to finding the best response against a fixed meta-strategy at every epoch. In this work, we propose Efficient PSRO (EPSRO) that largely improves the efficiency of the above two steps. Central to our development is the newly-introduced subroutine of no-regret optimization on the unrestricted-restricted (URR) game. By solving URR at each epoch, one can evaluate the current game and compute the best response in one forward pass without the need for meta-game simulations. Theoretically, we prove that the solution procedures of EPSRO offer a monotonic improvement on the exploitability, which none of existing PSRO methods possess. Furthermore, we prove that the no-regret optimization has a regret bound of $\mathcal{O}(\sqrt{T\log{[(k^2+k)/2]}})$, where $k$ is the size of restricted policy set. Most importantly, a desirable property of EPSRO is that it is parallelizable, this allows for highly efficient exploration in the policy space that induces behavioral diversity. We test EPSRO on three classes of games, and report a 50x speedup in wall-time and 10x data efficiency while maintaining similar exploitability as existing PSRO methods on Kuhn and Leduc Poker games.
The Alan Turing Institute（阿兰·图灵研究所）；Carnegie Mellon University（卡内基梅隆大学）；Delft University of Technology（代尔夫特理工大学）；Eindhoven University of Technology（埃因霍温理工大学）；University of Liverpool（利物浦大学）
UC-PSRO: Utility-Conditioned Policy-Space Response Oracles with a Communication-Dropout Curriculum for Game-Theoretic Course-of-Action Generation in Adversarial Swarms
Double Oracle Neural Architecture Search for Game Theoretic Deep Learning Models
Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局信息通信研究院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；KTH Royal Institute of Technology（瑞典皇家理工学院）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）；A*STAR Centre for Frontier AI Research（新加坡科技研究局前沿人工智能研究中心）
CAP-DO: Learned Contextual Action Proposals for Certified Double-Oracle Solving Across Related Zero-Sum Games
Hefei University of Technology（合肥工业大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；University of Science and Technology of China（中国科学技术大学）
