Robo2VLM：来自大规模野外机器人操作数据集的视觉问答
Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
AI总结：
提出Robo2VLM框架，利用大规模野外机器人轨迹数据生成VQA数据集Robo2VLM-1，以增强和评估VLM的空间与交互推理能力。
摘要：
视觉语言模型（VLM）通过互联网规模的图像-文本语料库获取真实世界知识和通用推理能力。它们可以通过场景理解和任务规划增强机器人系统，并辅助在机器人轨迹数据上训练出的视觉运动策略。我们探索相反的范式——利用丰富、真实、多模态的机器人轨迹数据来增强和评估VLM。在本文中，我们提出了Robo2VLM，一个用于VLM的视觉问答（VQA）数据集生成框架。给定一个人工遥操作的机器人轨迹，Robo2VLM从非视觉和非描述性的感知模态（如末端执行器位姿、夹爪开度和力传感）中推导出真实标签。基于这些模态，它将机器人轨迹分割成一系列操作阶段。在每个阶段，Robo2VLM利用场景和交互理解来识别机器人、任务目标和目标物体的3D属性。这些属性用于基于空间、目标条件和交互推理的问题模板，生成代表性的VQA查询——带有文本多项选择问题的图像。我们策划了Robo2VLM-1，一个大规模野外数据集，包含684,710个问题，覆盖463个不同场景和来自176k条真实机器人轨迹的3,396个机器人操作任务。结果表明，Robo2VLM-1可以基准测试并提升VLM在空间和交互推理方面的能力。
英文摘要：
Vision-Language Models (VLMs) acquire real-world knowledge and general reasoning ability through Internet-scale image-text corpora. They can augment robotic systems with scene understanding and task planning, and assist visuomotor policies that are trained on robot trajectory data. We explore the reverse paradigm - using rich, real, multi-modal robot trajectory data to enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual Question Answering (VQA) dataset generation framework for VLMs. Given a human tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual and non-descriptive sensory modalities, such as end-effector pose, gripper aperture, and force sensing. Based on these modalities, it segments the robot trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses scene and interaction understanding to identify 3D properties of the robot, task goal, and the target object. The properties are used to generate representative VQA queries - images with textural multiple-choice questions - based on spatial, goal-conditioned, and interaction reasoning question templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710 questions covering 463 distinct scenes and 3,396 robotic manipulation tasks from 176k real robot trajectories. Results suggest that Robo2VLM-1 can benchmark and improve VLM capabilities in spatial and interaction reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
