arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.01302cs.LG
收藏

基于遗憾的环境设计演化课程

Evolving Curricula with Regret-Based Environment Design

Meta AI;University of Oxford(牛津大学);UCL(伦敦大学学院);UC Berkeley(加州大学伯克利分校)

Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, Tim Rocktäschel

AI总结:

本文提出ACCEL,将演化式关卡编辑与基于遗憾的环境设计结合,在智能体能力边界上持续生成由简到繁的课程,并在多样环境中取得显著实证增益。

摘要:

使用强化学习(RL)训练具备通用能力的智能体仍是一项重大挑战。提升RL智能体鲁棒性的一条有前景途径是使用课程。其中一类方法将环境设计构建为学生与教师之间的博弈,采用基于遗憾的目标,在学生智能体能力边界上生成环境实例(即关卡)。这些方法具有通用性优势,并在均衡状态下具备理论保证,但在具有挑战性的设计空间中往往难以找到有效关卡。相比之下,演化方法试图渐进式改变环境复杂度,从而实现潜在的开放式学习,但通常依赖特定领域的启发式规则和大量计算资源。本文提出在有原则的、基于遗憾的课程中利用演化的力量。我们的方法称为通过编辑关卡进行对抗性复杂度累积(Adversarially Compounding Complexity by Editing Levels,ACCEL),旨在持续在智能体能力边界上生成关卡,使课程从简单开始并逐渐变得复杂。ACCEL保留了以往基于遗憾方法的理论优势,同时在多种不同环境中带来显著的实验性能提升。本文的交互式版本可在accelagent.github.io获取。

英文摘要:

It remains a significant challenge to train generally capable agents with reinforcement learning (RL). A promising avenue for improving the robustness of RL agents is through the use of curricula. One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities. These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces. By contrast, evolutionary approaches seek to incrementally alter environment complexity, resulting in potentially open-ended learning, but often rely on domain-specific heuristics and vast amounts of computational resources. In this paper we propose to harness the power of evolution in a principled, regret-based curriculum. Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical benefits of prior regret-based methods, while providing significant empirical gains in a diverse set of environments. An interactive version of the paper is available at accelagent.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑