深度强化学习真的是Atari上的超人吗?公平的竞争环境
Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
摘要:
深度强化学习(DRL)的一致性和可重复性评估并不简单。在街机学习环境(ALE)中,环境参数(如随机性或最大允许播放时间)的微小变化可能导致非常不同的性能。在这项工作中,我们讨论了比较不同的代理培训ALE的困难。为了进一步实现可重复和可比的DRL,我们引入了SABER,这是一种用于一般强化学习算法的标准化Atari BEnchmark。我们的方法扩展了以前的建议,并包含一套完整的环境参数以及训练和测试程序。然后,我们使用SABER来评估当前最先进的Rainbow。此外,我们引入了人类世界纪录的基线,并认为以前的专家或超人的DRL性能的索赔可能是不准确的。最后,我们提出了Rainbow-IQN,通过使用隐式分位数网络(IQN)扩展Rainbow,从而获得新的最先进的性能。源代码可用于再现性。
英文摘要:
Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade Learning Environment (ALE), small changes in environment parameters such as stochasticity or the maximum allowed play time can lead to very different performance. In this work, we discuss the difficulties of comparing different agents trained on ALE. In order to take a step further towards reproducible and comparable DRL, we introduce SABER, a Standardized Atari BEnchmark for general Reinforcement learning algorithms. Our methodology extends previous recommendations and contains a complete set of environment parameters as well as train and test procedures. We then use SABER to evaluate the current state of the art, Rainbow. Furthermore, we introduce a human world records baseline, and argue that previous claims of expert or superhuman performance of DRL might not be accurate. Finally, we propose Rainbow-IQN by extending Rainbow with Implicit Quantile Networks (IQN) leading to new state-of-the-art performance. Source code is available for reproducibility.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。