增强型POET：通过无限发明学习挑战及其解决方案的开放式强化学习
Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
摘要：
创建开放式算法，产生自己永无止境的新颖和适当的挑战性学习机会，可以帮助自动化和加速机器学习的进展。最近在这个方向上迈出的一步是成对开放式开拓者（POET），这是一种生成和解决自己的挑战的算法，并允许解决方案在挑战之间进行目标切换以避免局部最优。然而，原来的POET是无法证明其充分的创造性潜力，因为算法本身的局限性，因为外部的问题，包括有限的问题空间和缺乏一个普遍的进步措施。重要的是，这两种限制不仅对POET构成了障碍，而且对追求开放性也构成了障碍。在这里，我们介绍和经验验证两个新的创新，原来的算法，以及两个外部的创新，旨在帮助阐明其全部潜力。这四项进步共同促成了迄今为止最开放的算法演示。算法创新是（1）一个领域的一般措施，如何有意义的新挑战，使系统有可能创造和解决有趣的挑战无休止地，和（2）一个有效的启发式确定代理何时应该目标切换从一个问题到另一个（帮助开放式搜索更好的规模）。在算法本身之外，为了能够更明确地证明开放性，我们引入了（3）一种新颖的，更灵活的方式来编码环境挑战，以及（4）一个系统继续表现出开放性创新程度的通用度量。增强的POET产生了各种各样的复杂行为，解决了广泛的环境挑战，其中许多无法通过其他手段解决。
英文摘要：
Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation. Enhanced POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved through other means.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
