ManipVQA：将机器人可供性与物理基础信息注入多模态大语言模型
ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models
AI总结：
针对多模态大语言模型在机器人操作中缺乏可供性与物理概念理解的问题，提出ManipVQA框架，通过统一的VQA格式与微调策略注入机器人特定知识，在保留视觉推理能力的同时提升了操作任务表现。
摘要：
尽管将多模态大语言模型（MLLMs）与机器人系统集成已显著提升了机器人理解和执行自然语言指令的能力，但由于缺乏机器人领域的特定知识，其在操作任务中的表现仍受限。传统MLLMs通常在通用图文对上训练，导致其缺乏对操作至关重要的可供性和物理概念的理解。为弥补此差距，我们提出ManipVQA，一种通过视觉问答（VQA）格式向MLLMs注入以操作为中心知识的新框架。该方法涵盖工具检测、可供性识别及更广泛的物理概念理解。我们整理了包含交互物体的多样化图像数据集，以挑战机器人在工具检测、可供性预测和物理概念理解方面的能力。为有效整合该机器人特定知识与MLLMs固有的视觉推理能力，我们采用统一的VQA格式并设计微调策略。该策略在保留原有视觉推理能力的同时，融入了新获取的机器人见解。在机器人模拟器和各种视觉任务基准上的实证评估表明，ManipVQA表现出稳健的性能。代码和数据集已公开于https://github.com/SiyuanHuang95/ManipVQA。
英文摘要：
While the integration of Multi-modal Large Language Models (MLLMs) with robotic systems has significantly improved robots' ability to understand and execute natural language instructions, their performance in manipulation tasks remains limited due to a lack of robotics-specific knowledge. Conventional MLLMs are typically trained on generic image-text pairs, leaving them deficient in understanding affordances and physical concepts crucial for manipulation. To address this gap, we propose ManipVQA, a novel framework that infuses MLLMs with manipulation-centric knowledge through a Visual Question-Answering (VQA) format. This approach encompasses tool detection, affordance recognition, and a broader understanding of physical concepts. We curated a diverse dataset of images depicting interactive objects, to challenge robotic understanding in tool detection, affordance prediction, and physical concept comprehension. To effectively integrate this robotics-specific knowledge with the inherent vision-reasoning capabilities of MLLMs, we leverage a unified VQA format and devise a fine-tuning strategy. This strategy preserves the original vision-reasoning abilities while incorporating the newly acquired robotic insights. Empirical evaluations conducted in robotic simulators and across various vision task benchmarks demonstrate the robust performance of ManipVQA. The code and dataset are publicly available at https://github.com/SiyuanHuang95/ManipVQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
