Vid2Robot：基于交叉注意力Transformer的端到端视频条件策略学习
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
AI总结：
提出Vid2Robot，一种端到端的视频条件机器人策略，通过交叉注意力Transformer融合人类视频特征与机器人状态生成动作，并在真实机器人上取得显著效果与跨物体迁移能力。
摘要：
大规模多任务机器人操作系统通常依赖文本指定任务。在本研究中，我们探讨机器人能否通过观察人类来学习。为此，机器人必须理解人的意图，并在具身和环境存在差异的情况下执行推断出的任务。我们提出了Vid2Robot，一种端到端的视频条件策略，它将演示操作任务的人类视频作为输入，并输出机器人动作。我们的模型使用包含提示视频-机器人轨迹对的大规模数据集进行训练，以从视频中学习人类和机器人动作的统一表示。Vid2Robot在视频特征与当前机器人状态之间使用交叉注意力Transformer层，以产生动作并执行视频中展示的相同任务。我们使用辅助对比损失来对齐提示和机器人视频表示，以获得更好的策略。我们在真实世界机器人上评估了Vid2Robot，观察到使用人类提示视频时，相比BC-Z有超过20%的提升。此外，我们还展示了跨物体运动迁移能力，使视频条件策略能够将提示视频中在一个物体上观察到的运动迁移到机器人自身环境中的另一个物体上。视频可在https://vid2robot.github.io获取
英文摘要：
Large-scale multi-task robotic manipulation systems often rely on text to specify the task. In this work, we explore whether a robot can learn by observing humans. To do so, the robot must understand a person's intent and perform the inferred task despite differences in the embodiments and environments. We introduce Vid2Robot, an end-to-end video-conditioned policy that takes human videos demonstrating manipulation tasks as input and produces robot actions. Our model is trained with a large dataset of prompt video-robot trajectory pairs to learn unified representations of human and robot actions from videos. Vid2Robot uses cross-attention transformer layers between video features and the current robot state to produce the actions and perform the same task as shown in the video. We use auxiliary contrastive losses to align the prompt and robot video representations for better policies. We evaluate Vid2Robot on real-world robots and observe over 20% improvement over BC-Z when using human prompt videos. Further, we also show cross-object motion transfer ability that enables video-conditioned policies to transfer a motion observed on one object in the prompt video to another object in the robot's own environment. Videos available at https://vid2robot.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
