LUMOS：基于世界模型的语言条件模仿学习
LUMOS: Language-Conditioned Imitation Learning with World Models
AI总结：
本文提出 LUMOS 框架，在离线世界模型潜在空间中结合语言与图像目标重标注及内在奖励进行长时域模仿学习，以不到1%语言标注实现真实机器人零样本语言控制，并在 CALVIN 上超越同类方法。
摘要：
我们提出了 LUMOS，这是一种面向机器人的语言条件多任务模仿学习框架。LUMOS 通过在所学世界模型的潜在空间中进行大量长时域推演来练习技能，并将这些技能零样本迁移到真实机器人上。通过在所学世界模型的潜在空间中进行同策略学习，我们的算法缓解了大多数离线模仿学习方法所面临的策略诱导分布偏移问题。LUMOS 从非结构化游戏数据中学习，这些数据只有不到 1% 的事后语言标注，但在测试时可由语言命令进行引导。我们通过在训练期间将潜在规划与基于图像和基于语言的事后目标重标注相结合，并在多个时间步上优化世界模型潜在空间中定义的内在奖励，有效减少协变量偏移，从而实现连贯的长时域性能。在困难的长时域 CALVIN 基准实验中，LUMOS 在链式多任务评估上优于以往采用可比方法的基于学习的方法。据我们所知，我们首次在离线世界模型中为真实世界机器人学习语言条件的连续视觉运动控制。视频、数据集和代码可在 http://lumos.cs.uni-freiburg.de 获取。
英文摘要：
We introduce LUMOS, a language-conditioned multi-task imitation learning framework for robotics. LUMOS learns skills by practicing them over many long-horizon rollouts in the latent space of a learned world model and transfers these skills zero-shot to a real robot. By learning on-policy in the latent space of the learned world model, our algorithm mitigates policy-induced distribution shift which most offline imitation learning methods suffer from. LUMOS learns from unstructured play data with fewer than 1% hindsight language annotations but is steerable with language commands at test time. We achieve this coherent long-horizon performance by combining latent planning with both image- and language-based hindsight goal relabeling during training, and by optimizing an intrinsic reward defined in the latent space of the world model over multiple time steps, effectively reducing covariate shift. In experiments on the difficult long-horizon CALVIN benchmark, LUMOS outperforms prior learning-based methods with comparable approaches on chained multi-task evaluations. To the best of our knowledge, we are the first to learn a language-conditioned continuous visuomotor control for a real-world robot within an offline world model. Videos, dataset and code are available at http://lumos.cs.uni-freiburg.de.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
