arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.02439cs.LGcs.AI
收藏

重放引导的对抗环境设计

Replay-Guided Adversarial Environment Design

UCL(伦敦大学学院);FAIR(Facebook人工智能研究院);UC Berkeley(加州大学伯克利分校);University of Oxford(牛津大学)

Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, Tim Rocktäschel

摘要:

深度强化学习(RL)代理可以成功地推广到新的设置,如果在一个适当的不同的环境和任务配置集的训练。无监督环境设计(UED)是一种很有前途的自监督强化学习范式,其中未指定环境的自由参数在训练过程中自动适应智能体的能力,从而导致各种训练环境的出现。在这里,我们将优先级重放(PLR)作为UED,PLR是一种经验上成功但理论上没有动机的方法,它选择性地对随机生成的训练水平进行采样。我们认为,通过策划完全随机的水平,PLR,也可以产生新的和复杂的水平,有效的培训。这种见解揭示了一种自然的UED方法,我们称之为双课程设计(DCD)。至关重要的是,DCD包括PLR和流行的UED算法PAIRED作为特例,并继承了类似的理论保证。这种联系使我们能够开发新的理论PLR,提供一个版本的鲁棒性保证在纳什均衡。此外,我们的理论对PLR提出了一个非常违反直觉的改进:通过阻止智能体在未经策划的水平上更新其策略(在更少的数据上进行训练),我们可以提高到纳什均衡的收敛性。事实上,我们的实验证实,我们的新方法PLR $^{\perp}$在一系列分布外、零发射(zero-shot)传输任务上获得了更好的结果,此外还证明PLR $^{\perp}$提高了PAIRED的性能,它继承了PAIRED的理论框架。

英文摘要:

Deep reinforcement learning (RL) agents may successfully generalize to new settings if trained on an appropriately diverse set of environment and task configurations. Unsupervised Environment Design (UED) is a promising self-supervised RL paradigm, wherein the free parameters of an underspecified environment are automatically adapted during training to the agent's capabilities, leading to the emergence of diverse training environments. Here, we cast Prioritized Level Replay (PLR), an empirically successful but theoretically unmotivated method that selectively samples randomly-generated training levels, as UED. We argue that by curating completely random levels, PLR, too, can generate novel and complex levels for effective training. This insight reveals a natural class of UED methods we call Dual Curriculum Design (DCD). Crucially, DCD includes both PLR and a popular UED algorithm, PAIRED, as special cases and inherits similar theoretical guarantees. This connection allows us to develop novel theory for PLR, providing a version with a robustness guarantee at Nash equilibria. Furthermore, our theory suggests a highly counterintuitive improvement to PLR: by stopping the agent from updating its policy on uncurated levels (training on less data), we can improve the convergence to Nash equilibria. Indeed, our experiments confirm that our new method, PLR$^{\perp}$, obtains better results on a suite of out-of-distribution, zero-shot transfer tasks, in addition to demonstrating that PLR$^{\perp}$ improves the performance of PAIRED, from which it inherited its theoretical framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑