通过MetaVQA实现视觉语言模型的具身场景理解
Embodied Scene Understanding for Vision Language Models via MetaVQA
AI总结：
针对视觉语言模型缺乏具身场景理解评估基准的问题，提出MetaVQA综合基准，通过自动生成的交通场景问答对微调模型，提升其空间推理与安全驾驶能力，且具备仿真到真实的强迁移性。
摘要：
视觉语言模型（VLM）作为具身AI智能体在各类移动应用中展现出巨大潜力，但目前缺乏用于评估其空间推理与序列决策能力的标准化闭环基准。为解决这一问题，我们提出MetaVQA：这是一个综合性基准，旨在通过视觉问答（VQA）和闭环仿真，评估并增强VLM对空间关系与场景动态的理解能力。MetaVQA利用Set-of-Mark提示策略，结合nuScenes与Waymo数据集的俯视视角真值标注，基于多样化的真实交通场景自动生成大量问答对，确保指令以物体为中心且包含丰富上下文信息。我们的实验表明，使用MetaVQA数据集微调VLM，可显著提升其在安全关键仿真场景中的空间推理与具身场景理解能力，这不仅体现在VQA准确率的提升上，还体现在涌现出的具备安全意识的驾驶操作中。此外，该学习过程展现出从仿真到真实场景观测的强迁移性。代码与数据将在https://metadriverse.github.io/metavqa 公开。
英文摘要：
Vision Language Models (VLMs) demonstrate significant potential as embodied AI agents for various mobility applications. However, a standardized, closed-loop benchmark for evaluating their spatial reasoning and sequential decision-making capabilities is lacking. To address this, we present MetaVQA: a comprehensive benchmark designed to assess and enhance VLMs' understanding of spatial relationships and scene dynamics through Visual Question Answering (VQA) and closed-loop simulations. MetaVQA leverages Set-of-Mark prompting and top-down view ground-truth annotations from nuScenes and Waymo datasets to automatically generate extensive question-answer pairs based on diverse real-world traffic scenarios, ensuring object-centric and context-rich instructions. Our experiments show that fine-tuning VLMs with the MetaVQA dataset significantly improves their spatial reasoning and embodied scene comprehension in safety-critical simulations, evident not only in improved VQA accuracies but also in emerging safety-aware driving maneuvers. In addition, the learning demonstrates strong transferability from simulation to real-world observation. Code and data will be publicly available at https://metadriverse.github.io/metavqa .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
