面向机器人操作的物理扎根视觉-语言模型
Physically Grounded Vision-Language Models for Robotic Manipulation
AI总结：
针对 VLM 缺乏物体物理概念理解的问题，本文构建 PhysObjects 数据集并微调物理扎根 VLM，结合 LLM 规划器提升机器人操作规划与真实任务成功率。
摘要：
视觉-语言模型（VLM）的最新进展提升了其在视觉问答和图像描述等任务上的性能。因此，这些模型如今已具备对物理世界进行推理的良好条件，尤其是在机器人操作等领域。然而，当前 VLM 对常见物体的物理概念（例如材料、易碎性）理解有限，这限制了它们在需要与这类物体交互并进行物理推理的机器人操作任务中的实用性。为解决这一局限，我们提出 PhysObjects，这是一个以物体为中心的数据集，包含关于常见家居物体的 39.6K 条众包物理概念标注和 417K 条自动化物理概念标注。我们证明，在 PhysObjects 上微调 VLM，可通过从视觉外观中捕捉人类对这些概念的先验，提升其对物体物理概念的理解，包括对保留概念的泛化。我们将这种物理扎根 VLM 纳入一个交互式框架，该框架结合基于大语言模型的机器人规划器；结果表明，与未利用物理扎根 VLM 的基线相比，其在需要推理物体物理概念的任务上取得了更好的规划性能。此外，我们还在真实机器人上展示了物理扎根 VLM 的优势：它提高了任务成功率。我们发布了该数据集，并在 https://iliad.stanford.edu/pg-vlm/ 提供更多细节和结果可视化。
英文摘要：
Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. To address this limitation, we propose PhysObjects, an object-centric dataset of 39.6K crowd-sourced and 417K automated physical concept annotations of common household objects. We demonstrate that fine-tuning a VLM on PhysObjects improves its understanding of physical object concepts, including generalization to held-out concepts, by capturing human priors of these concepts from visual appearance. We incorporate this physically grounded VLM in an interactive framework with a large language model-based robotic planner, and show improved planning performance on tasks that require reasoning about physical object concepts, compared to baselines that do not leverage physically grounded VLMs. We additionally illustrate the benefits of our physically grounded VLM on a real robot, where it improves task success rates. We release our dataset and provide further details and visualizations of our results at https://iliad.stanford.edu/pg-vlm/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
