将视频模型转化为通用机器人策略
Turning Video Models into Generalist Robot Policies
AI总结：
提出一种解耦的视频到动作策略VERA，利用无动作视频世界模型和基于机器人雅可比矩阵的逆动力学模型，实现跨本体的零样本机器人控制。
摘要：
视频生成模型已成为一种有前景的机器人骨干网络，能够生成描绘跨本体和环境完成复杂任务的视频。最近的工作提出了机器人基础模型，通过使用带有动作标签的数据微调视频模型，联合预测未来观测和动作。在本文中，我们测试了一种替代方法的极限：保持视频规划器不变，同时训练一个特定本体的逆动力学模型（IDM）。这种解耦带来了几个自然的好处：视频规划器保持本体无关，不同的视频模型可以轻松互换而无需重新训练IDM，并且IDM可以独立地使用现成的自对弈数据进行训练。我们提出了一种闭环的视频到动作策略，该策略将无动作视频世界模型与基于机器人本体雅可比矩阵的精心设计的IDM相结合。我们证明了我们的IDM设计既数据高效又可扩展到高维动作空间。我们将该策略命名为视频到具身机器人动作模型（VERA），在模拟和真实世界基准测试中取得了强劲的性能，包括零样本的Panda机械臂操作和16自由度Allegro灵巧手立方体重新定向。通过将相同的视频规划器与不同的本体特定IDM配对，可以在多个本体上使用。我们的结果表明，解耦的视频规划加上忠实的视频到动作翻译是实现零样本、跨本体和可泛化机器人控制的可行替代途径。更多结果请访问我们的项目网站：https://vera.csail.mit.edu。
英文摘要：
Video generative models have emerged as a promising robotics backbone, capable of generating videos that depict the completion of complex tasks across embodiments and environments. Recent work proposes robot foundation models that jointly predict future observations and actions by finetuning video models with action-labeled data. In this paper, we test the limits of an alternative approach: leave the video planner as-is while training an embodiment-specific inverse dynamics model (IDM). This decoupling offers several natural benefits: the video planner remains embodiment-agnostic, different video models can be interchanged easily without re-training the IDM, and the IDM can be independently trained with readily available self-play data. We present a closed-loop, video-to-action policy that combines an action-free video world model with a carefully-designed IDM based on the robot embodiment Jacobian. We demonstrate that our IDM design is both data-efficient and scalable to high-dimensional action spaces. Our policy, which we coin the Video-to-Embodied Robot Action Model (VERA), achieves strong performance across simulated and real-world benchmarks, including zero-shot Panda arm manipulation and 16-DoF Allegro-hand dexterous cube re-orientation. The same video planner can be used across multiple embodiments by pairing it with different embodiment-specific IDMs. Our results show that decoupled video planning plus faithful video-to-action translation is a viable alternative route towards zero-shot, cross-embodiment, and generalizable robot control. More results are available on our project website: https://vera.csail.mit.edu.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
