Phantom: 仅使用人类视频训练机器人,无需机器人
Phantom: Training Robots Without Robots Using Only Human Videos
AI总结:
提出一种仅从人类视频演示中训练机器人操作策略的框架,通过手部姿态估计和视觉数据编辑将人类演示转化为机器人兼容的观察-动作对,实现零样本部署并达到最高92%的成功率。
摘要:
训练通用机器人需要从大规模且多样化的数据源中学习。当前方法严重依赖难以扩展的遥操作演示。我们提出一个可扩展的框架,可直接从人类视频演示中训练操作策略,无需任何机器人数据。我们的方法利用手部姿态估计和视觉数据编辑,将人类演示转化为机器人兼容的观察-动作对。我们修复人类手臂并叠加渲染的机器人以对齐视觉域。这使得无需任何微调即可在真实硬件上实现零样本部署。我们在包括可变形物体操作、多物体清扫和插入等一系列任务上展示了高达92%的强成功率。我们的方法可泛化到新环境并支持闭环执行。通过证明仅使用人类视频即可训练有效策略,我们的方法拓宽了可扩展机器人学习的路径。
英文摘要:
Training general-purpose robots requires learning from large and diverse data sources. Current approaches rely heavily on teleoperated demonstrations which are difficult to scale. We present a scalable framework for training manipulation policies directly from human video demonstrations, requiring no robot data. Our method converts human demonstrations into robot-compatible observation-action pairs using hand pose estimation and visual data editing. We inpaint the human arm and overlay a rendered robot to align the visual domains. This enables zero-shot deployment on real hardware without any fine-tuning. We demonstrate strong success rates-up to 92%-on a range of tasks including deformable object manipulation, multi-object sweeping, and insertion. Our approach generalizes to novel environments and supports closed-loop execution. By demonstrating that effective policies can be trained using only human videos, our method broadens the path to scalable robot learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。