以人类演示视频为提示学习可泛化的机器人策略
Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt
AI总结：
提出两阶段框架，利用人类演示视频学习可泛化机器人策略，无需遥操作数据或微调即可执行新任务。
摘要：
最近的机器人学习方法通常依赖于通过遥操作收集的大规模机器人数据集的模仿学习。面对新任务时，这些方法通常需要收集一组新的遥操作数据并微调策略。此外，遥操作数据收集流程也繁琐且昂贵。相反，人类能够通过观察他人操作高效学习新任务。在本文中，我们介绍了一种新颖的两阶段框架，利用人类演示学习可泛化的机器人策略。该策略可以直接以人类演示视频为提示，执行新任务，无需任何新的遥操作数据和模型微调。在第一阶段，我们训练视频生成模型，通过交叉预测捕获人类和机器人演示视频数据的联合表示。在第二阶段，我们使用新颖的原型对比损失将学习到的表示与人类和机器人之间的共享动作空间融合。在真实世界灵巧操作任务上的实证评估显示了所提出方法的有效性和泛化能力。
英文摘要：
Recent robot learning methods commonly rely on imitation learning from massive robotic dataset collected with teleoperation. When facing a new task, such methods generally require collecting a set of new teleoperation data and finetuning the policy. Furthermore, the teleoperation data collection pipeline is also tedious and expensive. Instead, human is able to efficiently learn new tasks by just watching others do. In this paper, we introduce a novel two-stage framework that utilizes human demonstrations to learn a generalizable robot policy. Such policy can directly take human demonstration video as a prompt and perform new tasks without any new teleoperation data and model finetuning at all. In the first stage, we train video generation model that captures a joint representation for both the human and robot demonstration video data using cross-prediction. In the second stage, we fuse the learned representation with a shared action space between human and robot using a novel prototypical contrastive loss. Empirical evaluations on real-world dexterous manipulation tasks show the effectiveness and generalization capabilities of our proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
