Autoverse：一种用于学习鲁棒具身智能体的可演化游戏语言
Autoverse: An Evolvable Game Language for Learning Robust Embodied Agents
AI总结：
本文提出可演化游戏语言 Autoverse，通过规则演化、搜索轨迹模仿学习与开放式 RL 课程生成，训练性能更强、更通用的具身玩家智能体。
摘要：
我们介绍了 Autoverse，这是一种用于单人二维网格游戏的、可演化的领域特定语言，并展示了其作为开放式学习（OEL）算法可扩展训练场地的用途。Autoverse 使用类似元胞自动机的重写规则来描述游戏机制，使其能够表达各种游戏环境（例如迷宫、地牢、推箱子谜题），这些环境都是强化学习（RL）智能体常用的测试平台。每条重写规则都可以表示为一系列简单的卷积，从而使环境能够在 GPU 上并行化，进而大幅加速 RL 训练。借助 Autoverse，我们提出通过从搜索中进行模仿学习来启动开放式学习。在这种方法中，我们首先演化 Autoverse 环境（包括其规则和初始地图拓扑），以最大化贪心树搜索发现新的最佳解所需的迭代次数，从而生成由复杂度不断提升的环境和游戏轨迹组成的课程。随后，我们使用模仿学习将这些专家游戏轨迹蒸馏到一个基于神经网络的策略中。最后，我们将学到的策略用作开放式 RL 的起点，在该过程中持续演化新的训练环境，以最大化 RL 玩家智能体的价值函数误差（作为其遗憾值或所生成环境可学习性的代理指标），并发现该方法提升了所得玩家智能体的性能和通用性。
英文摘要：
We introduce Autoverse, an evolvable, domain-specific language for single-player 2D grid-based games, and demonstrate its use as a scalable training ground for Open-Ended Learning (OEL) algorithms. Autoverse uses cellular-automaton-like rewrite rules to describe game mechanics, allowing it to express various game environments (e.g. mazes, dungeons, sokoban puzzles) that are popular testbeds for Reinforcement Learning (RL) agents. Each rewrite rule can be expressed as a series of simple convolutions, allowing for environments to be parallelized on the GPU, thereby drastically accelerating RL training. Using Autoverse, we propose jump-starting open-ended learning by imitation learning from search. In such an approach, we first evolve Autoverse environments (their rules and initial map topology) to maximize the number of iterations required by greedy tree search to discover a new best solution, producing a curriculum of increasingly complex environments and playtraces. We then distill these expert playtraces into a neural-network-based policy using imitation learning. Finally, we use the learned policy as a starting point for open-ended RL, where new training environments are continually evolved to maximize the RL player agent's value function error (a proxy for its regret, or the learnability of generated environments), finding that this approach improves the performance and generality of resultant player agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
