通过模仿生成视频实现机器人操作
Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations
AI总结：
本文提出RIGVid系统，通过模仿AI生成视频使机器人完成复杂操作任务，无需物理示范或特定机器人训练。系统利用视频扩散模型生成潜在示范视频，并通过视觉语言模型筛选符合指令的视频，再通过6D姿态跟踪器提取轨迹并映射到机器人。实验表明生成视频效果与真实示范相当，且性能随生成质量提升。
摘要：
本文介绍Robots Imitating Generated Videos (RIGVid)，一种使机器人通过模仿AI生成视频完成复杂操作任务（如倒水、擦拭和混合）的系统，无需任何物理示范或机器人特定训练。给定语言指令和初始场景图像，视频扩散模型生成潜在示范视频，视觉语言模型（VLM）自动筛选不符合指令的结果。6D姿态跟踪器从视频中提取物体轨迹，并以无关躯体的方式映射到机器人。通过广泛的实际评估，我们证明筛选后的生成视频与真实示范同样有效，且性能随生成质量提高。我们还表明依赖生成视频优于如使用VLM的关键点预测等更紧凑的替代方案，并且强大的6D姿态跟踪优于其他轨迹提取方法，如密集特征点跟踪。这些发现表明，由最新现成模型生成的视频可以为机器人操作提供有效的监督源。
英文摘要：
This work introduces Robots Imitating Generated Videos (RIGVid), a system that enables robots to perform complex manipulation tasks--such as pouring, wiping, and mixing--purely by imitating AI-generated videos, without requiring any physical demonstrations or robot-specific training. Given a language command and an initial scene image, a video diffusion model generates potential demonstration videos, and a vision-language model (VLM) automatically filters out results that do not follow the command. A 6D pose tracker then extracts object trajectories from the video, and the trajectories are retargeted to the robot in an embodiment-agnostic fashion. Through extensive real-world evaluations, we show that filtered generated videos are as effective as real demonstrations, and that performance improves with generation quality. We also show that relying on generated videos outperforms more compact alternatives such as keypoint prediction using VLMs, and that strong 6D pose tracking outperforms other ways to extract trajectories, such as dense feature point tracking. These findings suggest that videos produced by a state-of-the-art off-the-shelf model can offer an effective source of supervision for robotic manipulation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
