arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.01784cs.ROcs.CV
收藏

VILP:基于潜在视频规划的模仿学习

VILP: Imitation Learning with Latent Video Planning

School of Industrial Engineering, Purdue University(普渡大学工业工程学院);Elmore Family School of Electrical and Computer Engineering, Purdue University(普渡大学埃尔莫尔家族电气与计算机工程学院)

Zhengtong Xu, Qiang Qiu, Yu She

AI总结:

本文提出VILP方法,利用潜在视频扩散模型生成多视角时间一致的预测视频用于机器人模仿学习,在降低训练成本、提升推理速度与策略性能的同时减少对高质量动作数据的依赖。

摘要:

在生成式人工智能时代,将视频生成模型集成到机器人技术中为通用机器人智能体开辟了新的可能性。本文介绍了基于潜在视频规划的模仿学习(VILP)。我们提出了一种潜在视频扩散模型,用于生成具有良好时间一致性的预测性机器人视频。我们的方法能够从多个视角生成高度时间对齐的视频,这对机器人策略学习至关重要。我们的视频生成模型具有很高的时间效率。例如,它可以以5 Hz的速率从两个不同的视角生成视频,每个视频由六帧组成,分辨率为96x160像素。在实验中,我们证明了VILP在多个指标上优于现有的视频生成机器人策略:训练成本、推理速度、生成视频的时间一致性以及策略的性能。我们还将我们的方法与其他模仿学习方法进行了比较。我们的研究结果表明,VILP可以较少依赖大量高质量的任务特定机器人动作数据,同时仍保持稳健的性能。此外,VILP在表示多模态动作分布方面具有强大的能力。我们的论文提供了一个如何将视频生成模型有效集成到机器人策略中的实际示例,可能为相关领域和方向提供见解。更多细节请参阅我们的开源代码库 https://github.com/ZhengtongXu/VILP。

英文摘要:

In the era of generative AI, integrating video generation models into robotics opens new possibilities for the general-purpose robot agent. This paper introduces imitation learning with latent video planning (VILP). We propose a latent video diffusion model to generate predictive robot videos that adhere to temporal consistency to a good degree. Our method is able to generate highly time-aligned videos from multiple views, which is crucial for robot policy learning. Our video generation model is highly time-efficient. For example, it can generate videos from two distinct perspectives, each consisting of six frames with a resolution of 96x160 pixels, at a rate of 5 Hz. In the experiments, we demonstrate that VILP outperforms the existing video generation robot policy across several metrics: training costs, inference speed, temporal consistency of generated videos, and the performance of the policy. We also compared our method with other imitation learning methods. Our findings indicate that VILP can rely less on extensive high-quality task-specific robot action data while still maintaining robust performance. In addition, VILP possesses robust capabilities in representing multi-modal action distributions. Our paper provides a practical example of how to effectively integrate video generation models into robot policies, potentially offering insights for related fields and directions. For more details, please refer to our open-source repository https://github.com/ZhengtongXu/VILP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑