arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.02782cs.LGcs.AI
收藏

利用对抗性环境设计发现通用强化学习算法

Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design

University of Oxford(牛津大学);UCL(伦敦大学学院);Google DeepMind(谷歌DeepMind)

Matthew Thomas Jackson, Minqi Jiang, Jack Parker-Holder, Risto Vuorio, Chris Lu, Gregory Farquhar, Shimon Whiteson, Jakob Nicolaus Foerster

AI总结:

针对元学习RL更新规则在未知环境中的泛化差距,本文基于UED提出GROOVE,通过最大化元学习优化器的算法遗憾自动生成对抗课程,显著提升了相对LPG的泛化能力。

摘要:

过去十年,在人类研究人员手动设计的算法推动下,深度强化学习(RL)取得了巨大进展。近期研究表明,元学习更新规则是可行的,并有望发现在广泛RL任务上表现良好的算法。尽管Learned Policy Gradient(LPG)等算法已取得令人瞩目的初步成果,但当这些算法应用于未见过的环境时,仍存在泛化差距。在本研究中,我们考察了元训练分布的特征如何影响这些算法的泛化性能。受这一分析启发,并基于无监督环境设计(UED)的思想,我们提出了一种自动生成课程的新方法,用于最大化元学习优化器的遗憾;此外还提出了一种新的遗憾近似,我们将其命名为算法遗憾(AR)。由此得到的方法是General RL Optimizers Obtained Via Environment Design(GROOVE)。在一系列实验中,我们表明GROOVE相比LPG实现了更优的泛化,并将AR与UED中的基线指标进行评估,确认其是该场景下环境设计的关键组成部分。我们相信,该方法朝着发现能够解决广泛真实世界环境的真正通用RL算法迈出了一步。

英文摘要:

The past decade has seen vast progress in deep reinforcement learning (RL) on the back of algorithms manually designed by human researchers. Recently, it has been shown that it is possible to meta-learn update rules, with the hope of discovering algorithms that can perform well on a wide range of RL tasks. Despite impressive initial results from algorithms such as Learned Policy Gradient (LPG), there remains a generalization gap when these algorithms are applied to unseen environments. In this work, we examine how characteristics of the meta-training distribution impact the generalization performance of these algorithms. Motivated by this analysis and building on ideas from Unsupervised Environment Design (UED), we propose a novel approach for automatically generating curricula to maximize the regret of a meta-learned optimizer, in addition to a novel approximation of regret, which we name algorithmic regret (AR). The result is our method, General RL Optimizers Obtained Via Environment Design (GROOVE). In a series of experiments, we show that GROOVE achieves superior generalization to LPG, and evaluate AR against baseline metrics from UED, identifying it as a critical component of environment design in this setting. We believe this approach is a step towards the discovery of truly general RL algorithms, capable of solving a wide range of real-world environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑