VLM See, Robot Do: 通过视觉语言模型将人类演示视频转化为机器人动作规划
VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model
AI总结：
本研究提出SeeDo方法，利用视觉语言模型解释人类演示视频，通过关键帧选择、视觉感知和推理生成机器人任务规划，实验验证其优越性并成功部署于仿真和真实机器人。
摘要：
视觉语言模型（VLMs）近年来因其常识推理能力和泛化性而被应用于机器人领域。现有工作已应用VLM从自然语言指令生成任务与运动规划，并模拟训练数据以用于机器人学习。在本工作中，我们探索使用VLM来解释人类演示视频并生成机器人任务规划。我们的方法将关键帧选择、视觉感知和VLM推理整合到一个流程中。我们将其命名为SeeDo，因为它使VLM能够“看到”人类演示，并向机器人解释相应的计划以供其“执行”。为验证我们的方法，我们收集了一组长时程人类视频，展示了三个不同类别中的拾取与放置任务，并设计了一组指标，以全面地将SeeDo与包括最先进的视频输入VLM在内的多个基线进行基准测试。实验证明了SeeDo的优越性能。我们进一步在仿真环境和真实机器人手臂上部署了生成的任务规划。
英文摘要：
Vision Language Models (VLMs) have recently been adopted in robotics for their capability in common sense reasoning and generalizability. Existing work has applied VLMs to generate task and motion planning from natural language instructions and simulate training data for robot learning. In this work, we explore using VLM to interpret human demonstration videos and generate robot task planning. Our method integrates keyframe selection, visual perception, and VLM reasoning into a pipeline. We named it SeeDo because it enables the VLM to ''see'' human demonstrations and explain the corresponding plans to the robot for it to ''do''. To validate our approach, we collected a set of long-horizon human videos demonstrating pick-and-place tasks in three diverse categories and designed a set of metrics to comprehensively benchmark SeeDo against several baselines, including state-of-the-art video-input VLMs. The experiments demonstrate SeeDo's superior performance. We further deployed the generated task plans in both a simulation environment and on a real robot arm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
