学习面向未见操作任务的视频条件策略
Learning Video-Conditioned Policies for Unseen Manipulation Tasks
AI总结：
提出ViP方法，利用预训练视频嵌入将人类演示映射为机器人操作策略，实现零样本控制未见任务，并在多任务环境中超越现有技术水平。
摘要：
非专家用户指定机器人命令的能力对于构建能够解决大量任务的通用智能体至关重要。指定预期机器人目标的一种便捷方式是通过展示目标任务的人的视频。先前的工作通常旨在模仿在机器人环境中执行的人类演示，而这里我们关注一个更现实且更具挑战性的设置，即演示在自然且多样化的人类环境中记录。我们提出了视频条件策略学习（ViP），一种数据驱动的方法，将先前未见任务的人类演示映射到机器人操作技能。为此，我们学习我们的策略，根据当前场景观察和目标任务的视频生成适当的动作。为了鼓励对新任务的泛化，我们在训练期间避免特定任务，并从无标签的机器人轨迹和相应的机器人视频中学习我们的策略。我们框架中的机器人和人类视频均由为人类动作识别预训练的视频嵌入表示。在测试时，我们首先在公共视频嵌入空间中将人类视频翻译为机器人视频，然后使用生成的嵌入来条件化我们的策略。值得注意的是，我们的方法实现了通过人类演示以零样本方式控制机器人，即在训练期间不使用与人类指令配对的机器人轨迹。我们在具有挑战性的多任务机器人操作环境集上验证了我们的方法，并超越了现有技术水平。我们的方法还在一个新的具有挑战性的零样本设置中表现出色，该设置中训练期间不使用配对数据。
英文摘要：
The ability to specify robot commands by a non-expert user is critical for building generalist agents capable of solving a large variety of tasks. One convenient way to specify the intended robot goal is by a video of a person demonstrating the target task. While prior work typically aims to imitate human demonstrations performed in robot environments, here we focus on a more realistic and challenging setup with demonstrations recorded in natural and diverse human environments. We propose Video-conditioned Policy learning (ViP), a data-driven approach that maps human demonstrations of previously unseen tasks to robot manipulation skills. To this end, we learn our policy to generate appropriate actions given current scene observations and a video of the target task. To encourage generalization to new tasks, we avoid particular tasks during training and learn our policy from unlabelled robot trajectories and corresponding robot videos. Both robot and human videos in our framework are represented by video embeddings pre-trained for human action recognition. At test time we first translate human videos to robot videos in the common video embedding space, and then use resulting embeddings to condition our policies. Notably, our approach enables robot control by human demonstrations in a zero-shot manner, i.e., without using robot trajectories paired with human instructions during training. We validate our approach on a set of challenging multi-task robot manipulation environments and outperform state of the art. Our method also demonstrates excellent performance in a new challenging zero-shot setup where no paired data is used during training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
