在开放世界生存游戏Crafter中使用以对象为中心的Agent学习泛化
Learning to Generalize with Object-centric Agents in the Open World Survival Game Crafter
摘要：
强化学习代理必须泛化超出他们的训练经验。以前的工作主要集中在相同的培训和评价环境。从最近推出的工匠基准，2D开放世界生存游戏，我们介绍了一组新的环境，适合评估一些代理的能力，以前看不见的（数量）对象概括和快速适应（元学习）。在Crafter中，当训练1M步时，智能体通过解锁成就（例如收集资源）的数量进行评估。我们表明，目前的代理努力推广，并引入新的以对象为中心的代理，提高了强大的基线。我们还通过几个实验为Crafter的未来工作提供了普遍感兴趣的重要见解。我们表明，仔细的超参数调整提高了PPO基线代理的大幅度，甚至前馈代理可以解锁几乎所有的成就，依靠库存显示。我们在原有的Crafter环境中实现了最先进的性能。此外，当训练超过100万步时，我们的优化代理可以解锁几乎所有成就。我们表明，经常性的PPO代理改善前馈的，即使与库存信息删除。我们介绍CrafterOOD，一组15个新的环境，评估OOD泛化。在CrafterOOD上，我们发现当前的代理无法泛化，而我们新的以对象为中心的代理实现了最先进的OOD泛化，同时也是可解释的。代码是公开的。
英文摘要：
Reinforcement learning agents must generalize beyond their training experience. Prior work has focused mostly on identical training and evaluation environments. Starting from the recently introduced Crafter benchmark, a 2D open world survival game, we introduce a new set of environments suitable for evaluating some agent's ability to generalize on previously unseen (numbers of) objects and to adapt quickly (meta-learning). In Crafter, the agents are evaluated by the number of unlocked achievements (such as collecting resources) when trained for 1M steps. We show that current agents struggle to generalize, and introduce novel object-centric agents that improve over strong baselines. We also provide critical insights of general interest for future work on Crafter through several experiments. We show that careful hyper-parameter tuning improves the PPO baseline agent by a large margin and that even feedforward agents can unlock almost all achievements by relying on the inventory display. We achieve new state-of-the-art performance on the original Crafter environment. Additionally, when trained beyond 1M steps, our tuned agents can unlock almost all achievements. We show that the recurrent PPO agents improve over feedforward ones, even with the inventory information removed. We introduce CrafterOOD, a set of 15 new environments that evaluate OOD generalization. On CrafterOOD, we show that the current agents fail to generalize, whereas our novel object-centric agents achieve state-of-the-art OOD generalization while also being interpretable. Our code is public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
