University College London(伦敦大学学院);Shanghai Jiao Tong University(上海交通大学);Institute of Automation, CAS(中国科学院自动化研究所);University of California, Irvine(加利福尼亚大学尔湾分校);King’s College London(伦敦国王学院)
Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, Yaodong Yang
When solving two-player zero-sum games, multi-agent reinforcement learning (MARL) algorithms often create populations of agents where, at each iteration, a new agent is discovered as the best response to a mixture over the opponent population. Within such a process, the update rules of "who to compete with" (i.e., the opponent mixture) and "how to beat them" (i.e., finding best responses) are underpinned by manually developed game theoretical principles such as fictitious play and Double Oracle. In this paper, we introduce a novel framework -- Neural Auto-Curricula (NAC) -- that leverages meta-gradient descent to automate the discovery of the learning update rule without explicit human design. Specifically, we parameterise the opponent selection module by neural networks and the best-response module by optimisation subroutines, and update their parameters solely via interaction with the game engine, where both players aim to minimise their exploitability. Surprisingly, even without human design, the discovered MARL algorithms achieve competitive or even better performance with the state-of-the-art population-based game solvers (e.g., PSRO) on Games of Skill, differentiable Lotto, non-transitive Mixture Games, Iterated Matching Pennies, and Kuhn Poker. Additionally, we show that NAC is able to generalise from small games to large games, for example training on Kuhn Poker and outperforming PSRO on Leduc Poker. Our work inspires a promising future direction to discover general MARL algorithms solely from data.
Collaborative Auto-Curricula Multi-Agent Reinforcement Learning with Graph Neural Network Communication Layer for Open-ended Wildfire-Management Resource Distribution
Heterogeneous Adversarial Play in Interactive Environments
Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学);School of Psychological and Cognitive Sciences, Peking University(心理与认知科学学院,北京大学);School of Computer Science & Technology, Beijing Institute of Technology(计算机科学与技术学院,北京理工大学);State Key Lab of General AI, Peking University(通用人工智能国家重点实验室,北京大学);Beijing Key Laboratory of Behavior and Mental Health, Peking University(北京行为与心理健康重点实验室,北京大学);Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing(北京理工大学长江三角洲地区研究院,嘉兴);Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence(具身智能实验室,北京大学-武汉人工智能研究院)
Intelligent Autonomous Systems, Technische Universität Darmstadt(达姆施塔特工业大学智能自主系统(Intelligent Autonomous Systems, Technische Universität Darmstadt));Department of Electrical Engineering and Automation, Aalto University(阿尔托大学电气工程与自动化系(Department of Electrical Engineering and Automation, Aalto University))