从人类视频中学习可供性作为机器人的通用表征
Affordances from Human Videos as a Versatile Representation for Robotics
AI总结：
本文提出VRB方法，通过从互联网人类视频中学习视觉可供性模型，估计场景中人类可能的交互位置和方式，并将其无缝集成到离线模仿学习、探索、目标条件学习和强化学习动作参数化四种范式中，在多个真实环境和任务上验证了机器人执行复杂任务的有效性。
摘要：
构建一个能够通过观察人类来理解并学习交互的机器人，已经启发了多个视觉问题。然而，尽管在静态数据集上取得了一些成功的结果，目前仍不清楚如何将现有模型直接用于机器人。在本文中，我们旨在通过以环境为中心的方式利用人类交互视频来弥合这一差距。利用互联网上的人类行为视频，我们训练了一个视觉可供性模型，用于估计场景中人类可能在何处以及以何种方式进行交互。这些行为可供性的结构直接使机器人能够执行许多复杂任务。我们展示了如何将我们的可供性模型无缝集成到四种机器人学习范式中，包括离线模仿学习、探索、目标条件学习以及用于强化学习的动作参数化。我们在4个真实世界环境、超过10个不同任务以及2个在野外运行的机器人平台上展示了我们方法（我们称之为VRB）的有效性。结果、可视化和视频见 https://robo-affordances.github.io/
英文摘要：
Building a robot that can understand and learn to interact by watching humans has inspired several vision problems. However, despite some successful results on static datasets, it remains unclear how current models can be used on a robot directly. In this paper, we aim to bridge this gap by leveraging videos of human interactions in an environment centric manner. Utilizing internet videos of human behavior, we train a visual affordance model that estimates where and how in the scene a human is likely to interact. The structure of these behavioral affordances directly enables the robot to perform many complex tasks. We show how to seamlessly integrate our affordance model with four robot learning paradigms including offline imitation learning, exploration, goal-conditioned learning, and action parameterization for reinforcement learning. We show the efficacy of our approach, which we call VRB, across 4 real world environments, over 10 different tasks, and 2 robotic platforms operating in the wild. Results, visualizations and videos at https://robo-affordances.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
