Learning the behavior of large agent populations is an important task for numerous research areas. Although the field of multi-agent reinforcement learning (MARL) has made significant progress towards solving these systems, solutions for many agents often remain computationally infeasible and lack theoretical guarantees. Mean Field Games (MFGs) address both of these issues and can be extended to Graphon MFGs (GMFGs) to include network structures between agents. Despite their merits, the real world applicability of GMFGs is limited by the fact that graphons only capture dense graphs. Since most empirically observed networks show some degree of sparsity, such as power law graphs, the GMFG framework is insufficient for capturing these network topologies. Thus, we introduce the novel concept of Graphex MFGs (GXMFGs) which builds on the graph theoretical concept of graphexes. Graphexes are the limiting objects to sparse graph sequences that also have other desirable features such as the small world property. Learning equilibria in these games is challenging due to the rich and sparse structure of the underlying graphs. To tackle these challenges, we design a new learning algorithm tailored to the GXMFG setup. This hybrid graphex learning approach leverages that the system mainly consists of a highly connected core and a sparse periphery. After defining the system and providing a theoretical analysis, we state our learning approach and demonstrate its learning capabilities on both synthetic graphs and real-world networks. This comparison shows that our GXMFG learning algorithm successfully extends MFGs to a highly relevant class of hard, realistic learning problems that are not accurately addressed by current MARL and MFG methods.
Using Generative Adversarial Nets on Atari Games for Feature Extraction in Deep Reinforcement Learning
Middle East Technical University(中东技术大学)
深度强化学习(Deep Reinforcement Learning,DRL)已成功应用于机器人导航和自动视频游戏等多个研究领域。然而,这些方法需要过多的计算和与环境的交互,因此需要提高样本效率。这一要求的主要原因是稀疏和延迟的奖励不能为深度神经网络的表示学习提供有效的监督。在这项研究中,近端策略优化(PPO)算法与生成对抗网络(GANs)增强,通过强制网络学习有效的表示而不依赖于稀疏和延迟的奖励作为监督来提高样本效率。结果表明,通过联合训练DRL代理和GAN代理,可以获得更高的性能。- Derin Pekistirmeli Oakme,robot navigasyonu ve otomatikelestirilmis video oyunu oaku a gibi arastirma alanlarinda basariyla uygulanmaktadir.此外,我们还可以看到,由于气候变化和环境变化,我们的工作人员无法确定他们的工作是否正常。但是,在一个人的生活中,人们对自己的生活和生活的看法是,他们的生活和生活是如此的丰富多彩,以至于他们的生活和生活都变得如此的艰难。但据了解,Uretici Cekismeli Aglar(UCA)算法的优化政策方案将使目标明确,并使目标明确或使目标明确,从而使目标明确。其他的声音算法也可以用其他的方法来实现。
ARAC: Adaptive Regularized Multi-Agent Soft Actor-Critic in Graph-Structured Adversarial Games
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Dynamic one-time delivery of critical data by small and sparse UAV swarms: a model problem for MARL scaling studies
Swedish Defence Research Agency (FOI)(瑞典国防研究机构(FOI));Chalmers University of Technology(查尔姆斯理工大学);the University of Gothenburg, Department of Mathematical Sciences(哥德堡大学数学科学系)
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
Tsinghua University(清华大学);National University of Defense Technology(国防科技大学);RealAI(瑞莱智慧);Tencent AI Lab(腾讯人工智能实验室)
深度强化学习(DRL)在复杂的视频游戏上实现了超人的性能(例如,星际争霸II和Dota II)。然而,目前的DRL系统仍然受到多智能体协调,稀疏奖励,随机环境等的挑战。为了解决这些挑战,我们采用了足球视频游戏,例如,Google Research Football(GRF)作为我们的测试平台,开发了一个端到端的基于学习的AI系统(表示为TiKick)来完成这项具有挑战性的任务。在这项工作中,我们首先从联盟训练中获得的单智能体专家的自我比赛中生成了一个大型重放数据集。然后,我们开发了一个分布式学习系统和新的离线算法,从固定的单智能体数据集中学习强大的多智能体AI。据我们所知,Tikick是第一个基于学习的人工智能系统,可以接管多智能体谷歌研究足球完整游戏,而以前的工作可以控制单个智能体或在玩具学术场景上进行实验。大量的实验进一步表明,我们的预训练模型可以加速现代多智能体算法的训练过程,我们的方法在各种学术场景中达到了最先进的性能。
On Bonus-Based Exploration Methods in the Arcade Learning Environment
MILA, Université de Montréal(蒙特利尔大学米拉研究所);Google Brain(谷歌大脑)
应用于Atari 2600游戏的强化学习中的探索研究强调解决困难的探索问题,例如Montezuma的复仇(Bellemare et al.,2016)。最近,基于奖金的探索方法,通过增加环境奖励进行探索,在这些领域达到了高于人类平均水平的表现。在本文中,我们重新评估流行的奖金为基础的探索方法在一个共同的评估框架。我们结合彩虹(Hessel等人,2018年)与不同的探索奖金,并评估其性能蒙特祖玛的复仇,贝勒马尔等人。的一套艰苦的探索游戏与稀疏的奖励,和整个雅达利2600套件。我们发现,虽然探索奖金导致更高的分数蒙特祖玛的复仇,他们不提供有意义的收益比简单的$ε$贪婪的计划。事实上,我们发现,在该游戏上表现最好的方法往往不如$ε$-贪婪的容易探索雅达利2600游戏。我们发现,我们的结论仍然有效,即使超参数调整这些容易探索的游戏。最后,我们发现,没有一个调查的方法受益于额外的训练样本(10亿帧,而彩虹的2亿帧)Bellemare等人。的艰苦探索游戏。我们的研究结果表明,最近在蒙特祖玛的复仇的收益可能更好地归因于架构的变化,而不是更好的探索计划,并在雅达利2600游戏的探索研究的实际进展速度可能已经混淆了一个单一的域上的良好结果。
Approximate Feedback Nash Equilibria with Sparse Inter-Agent Dependencies
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin(德克萨斯大学奥斯汀分校奥登计算工程与科学研究所);Department of Electrical Engineering and Computer Sciences, University of California, Berkeley(加州大学伯克利分校电气工程与计算机科学系);DEVCOM Army Research Laboratory(DEVCOM陆军研究实验室)