arXiv 2410.117582025-05-15cs.ROcs.CLcs.CVcs.LG

从视频中进行潜在动作预训练

Latent Action Pretraining from Videos

KAIST（韩国科学技术院）；University of Washington（华盛顿大学）；Microsoft Research（微软研究院）；NVIDIA（英伟达）；Allen Institute for AI（人工智能算法研究所）

Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee… 展开作者

Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, Minjoon Seo

备注：: ICLR 2025 Website: https://latentactionpretraining.github.io

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LAPA无监督预训练方法，通过VQ-VAE式潜在动作量化从无动作标签视频学习VLA模型，并在小规模机器人数据上微调，显著提升真实操作泛化性能。

摘要：

我们提出面向通用动作模型的潜在动作预训练（Latent Action Pretraining for general Action models，LAPA），这是一种在没有真实机器人动作标签的情况下预训练视觉-语言-动作（Vision-Language-Action，VLA）模型的无监督方法。现有VLA模型在预训练期间通常需要由人类遥操作员收集的动作标签，这显著限制了可能的数据来源和规模。在本工作中，我们提出一种从没有机器人动作标签的互联网规模视频中学习的方法。我们首先训练一个动作量化模型，利用基于VQ-VAE的目标来学习图像帧之间的离散潜在动作；随后预训练一个潜在VLA模型，根据观测和任务描述预测这些潜在动作；最后在小规模机器人操作数据上微调VLA，以实现从潜在动作到机器人动作的映射。实验结果表明，我们的方法显著优于现有从大规模视频训练机器人操作策略的技术。此外，在需要语言条件化、对未见物体泛化以及对未见指令进行语义泛化的真实世界操作任务上，它优于使用机器人动作标签训练的最先进VLA模型。仅在人类操作视频上训练也表现出正向迁移，为利用网络规模数据构建机器人基础模型开辟了潜力。

英文摘要：

We introduce Latent Action Pretraining for general Action models (LAPA), an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. Existing Vision-Language-Action models require action labels typically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method to learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE-based objective to learn discrete latent actions between image frames, then pretrain a latent VLA model to predict these latent actions from observations and task descriptions, and finally finetune the VLA on small-scale robot manipulation data to map from latent to robot actions. Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the-art VLA model trained with robotic action labels on real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instructions. Training only on human manipulation videos also shows positive transfer, opening up the potential for leveraging web-scale data for robotics foundation model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。