让我展示给你看：通过从第一人称视频中检索来学习机器人操作
Let Me Show You: Learning by Retrieving from Egocentric Video for Robotic Manipulation
AI总结：
本文提出RfV方法，通过从人类第一人称视频库中检索任务相关演示并提取可供性掩码与手部轨迹等中层信息，结合视频检索器和策略生成器，使机器人能够学习并泛化到训练数据之外的操作任务，在模拟与真实环境中均显著优于传统系统。
摘要：
在复杂和不确定环境中运行的机器人面临着相当大的挑战。先进的机器人系统通常依赖大规模数据集来学习操作任务。相比之下，当人类面对不熟悉的任务（例如组装椅子）时，一种常见的方法是通过观看视频演示来学习。在本文中，我们提出了一种新颖的方法，即通过从视频中检索（Retrieving-from-Video，RfV）来学习机器人策略，利用人类演示中的类比来解决操作任务。我们的系统构建了一个视频库，其中包含人类执行各种日常任务的录像。为了丰富这些视频中的知识，我们提取中层信息，例如物体可供性掩码和手部运动轨迹，这些信息作为额外输入，以增强机器人模型的学习和泛化能力。我们进一步设计了一个双组件系统：一个视频检索器，它利用外部视频库根据任务规范获取与任务相关的视频；以及一个策略生成器，它将检索到的知识整合到学习循环中。这种方法使机器人能够针对各种场景制定自适应响应，并泛化到训练数据之外的任务。通过在多个模拟和真实世界环境中的严格测试，我们的系统展示出相较于传统机器人系统的显著性能提升，体现了机器人领域的一项重大突破。
英文摘要：
Robots operating in complex and uncertain environments face considerable challenges. Advanced robotic systems often rely on extensive datasets to learn manipulation tasks. In contrast, when humans are faced with unfamiliar tasks, such as assembling a chair, a common approach is to learn by watching video demonstrations. In this paper, we propose a novel method for learning robot policies by Retrieving-from-Video (RfV), using analogies from human demonstrations to address manipulation tasks. Our system constructs a video bank comprising recordings of humans performing diverse daily tasks. To enrich the knowledge from these videos, we extract mid-level information, such as object affordance masks and hand motion trajectories, which serve as additional inputs to enhance the robot model's learning and generalization capabilities. We further feature a dual-component system: a video retriever that taps into an external video bank to fetch task-relevant video based on task specification, and a policy generator that integrates this retrieved knowledge into the learning cycle. This approach enables robots to craft adaptive responses to various scenarios and generalize to tasks beyond those in the training data. Through rigorous testing in multiple simulated and real-world settings, our system demonstrates a marked improvement in performance over conventional robotic systems, showcasing a significant breakthrough in the field of robotics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
