DITTO：基于世界模型的离线模仿学习
DITTO: Offline Imitation Learning with World Models
AI总结：
针对模仿学习面临的高维观测、离线学习及协变量偏移问题，提出基于世界模型潜空间多步散度优化的DITTO算法，在Atari像素任务上取得SOTA性能与样本效率，且适配其他算法可提升其表现。
摘要：
要让模仿学习算法能应对现实世界的挑战，必须能够处理高维观测、离线学习以及策略引发的协变量偏移问题。我们提出了DITTO，一种可同时解决这三类问题的离线模仿学习算法。DITTO在已学习的世界模型的潜空间中优化一种新型距离度量：首先，我们利用所有可用的轨迹数据训练一个世界模型；随后，模仿智能体从专家起始状态在已学习的模型中展开，并因多时间步内与专家数据集的潜空间差异而受到惩罚。我们使用标准强化学习算法优化这种多步潜空间差异，该方法经理论证明可实现模仿学习，并且在无需任何在线环境访问的情况下，在一系列基于像素输入的Atari环境中取得了当前最优的性能与样本效率。我们还将其他标准模仿学习算法适配到世界模型框架下，结果表明这大幅提升了它们的性能。我们的研究结果表明，对世界模型的创新性运用可以构建出简单、鲁棒且高性能的策略学习框架。
英文摘要：
For imitation learning algorithms to scale to real-world challenges, they must handle high-dimensional observations, offline learning, and policy-induced covariate-shift. We propose DITTO, an offline imitation learning algorithm which addresses all three of these problems. DITTO optimizes a novel distance metric in the latent space of a learned world model: First, we train a world model on all available trajectory data, then, the imitation agent is unrolled from expert start states in the learned model, and penalized for its latent divergence from the expert dataset over multiple time steps. We optimize this multi-step latent divergence using standard reinforcement learning algorithms, which provably induces imitation learning, and empirically achieves state-of-the art performance and sample efficiency on a range of Atari environments from pixels, without any online environment access. We also adapt other standard imitation learning algorithms to the world model setting, and show that this considerably improves their performance. Our results show how creative use of world models can lead to a simple, robust, and highly-performant policy-learning framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
