arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.02745cs.AIcs.MA
收藏

神经自动课程

Neural Auto-Curricula

University College London(伦敦大学学院);Shanghai Jiao Tong University(上海交通大学);Institute of Automation, CAS(中国科学院自动化研究所);University of California, Irvine(加利福尼亚大学尔湾分校);King’s College London(伦敦国王学院)

Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, Yaodong Yang

摘要:

在解决两人零和游戏时,多智能体强化学习(MARL)算法通常会创建智能体群体,在每次迭代中,都会发现一个新的智能体作为对对手群体的最佳反应。在这样的过程中,“与谁竞争”的更新规则(即,对手混合物)和“如何击败他们”(即,寻找最佳对策)是由手动开发的游戏理论原则,如虚构的发挥和双甲骨文。在本文中,我们介绍了一种新的框架-神经自动课程(NAC)-利用Meta梯度下降自动发现的学习更新规则,而无需明确的人类设计。具体来说,我们参数化的对手选择模块的神经网络和最佳响应模块的优化子程序,并更新其参数仅通过与游戏引擎,其中两个球员的目标是尽量减少他们的剥削。令人惊讶的是,即使没有人类设计,所发现的MARL算法也实现了与最先进的基于人口的游戏求解器(例如,PSRO)的游戏技巧,可微乐透,非传递混合游戏,迭代匹配便士,和库恩扑克。此外,我们表明,NAC能够从小型游戏推广到大型游戏,例如Kuhn扑克的培训和Leduc扑克的PSRO。我们的工作启发了一个有前途的未来方向,发现一般的MARL算法完全从数据。

英文摘要:

When solving two-player zero-sum games, multi-agent reinforcement learning (MARL) algorithms often create populations of agents where, at each iteration, a new agent is discovered as the best response to a mixture over the opponent population. Within such a process, the update rules of "who to compete with" (i.e., the opponent mixture) and "how to beat them" (i.e., finding best responses) are underpinned by manually developed game theoretical principles such as fictitious play and Double Oracle. In this paper, we introduce a novel framework -- Neural Auto-Curricula (NAC) -- that leverages meta-gradient descent to automate the discovery of the learning update rule without explicit human design. Specifically, we parameterise the opponent selection module by neural networks and the best-response module by optimisation subroutines, and update their parameters solely via interaction with the game engine, where both players aim to minimise their exploitability. Surprisingly, even without human design, the discovered MARL algorithms achieve competitive or even better performance with the state-of-the-art population-based game solvers (e.g., PSRO) on Games of Skill, differentiable Lotto, non-transitive Mixture Games, Iterated Matching Pennies, and Kuhn Poker. Additionally, we show that NAC is able to generalise from small games to large games, for example training on Kuhn Poker and outperforming PSRO on Leduc Poker. Our work inspires a promising future direction to discover general MARL algorithms solely from data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑