以自我为中心的多视角场景中视觉-语言模型的空间推理
Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes
AI总结：
本文构建面向自我中心多视角户外场景的Ego3D-Bench评测16个VLM，并提出可模块化集成的Ego3D-VLM后训练框架，显著提升3D空间推理与距离估计。
摘要：
理解3D空间关系仍然是当前视觉-语言模型（VLM）的一个主要局限。先前工作通过创建基于单张图像或室内视频的空间问答（QA）数据集来解决这一问题。然而，机器人和自动驾驶汽车等真实世界具身AI智能体通常依赖以自我为中心的多视角观测。为此，我们推出Ego3D-Bench，这是一个旨在利用以自我为中心的多视角户外数据评估VLM空间推理能力的新基准。Ego3D-Bench包含8600多个问答对，由人工标注者深度参与创建，以确保质量和多样性。我们对16个SOTA VLM进行了基准测试，包括GPT-4o、Gemini1.5-Pro、InternVL3和Qwen2.5-VL。结果揭示了人类水平分数与VLM性能之间存在显著差距，凸显当前VLM仍未达到人类水平的空间理解能力。为弥合这一差距，我们提出Ego3D-VLM，这是一个增强VLM 3D空间推理能力的后训练框架。Ego3D-VLM基于估计的全局3D坐标生成认知地图，在多选QA上实现12%的平均提升，在绝对距离估计上实现56%的平均提升。Ego3D-VLM采用模块化设计，可与任何现有VLM集成。Ego3D-Bench和Ego3D-VLM共同为在真实世界多视角环境中迈向人类水平空间理解提供了宝贵工具。
英文摘要：
Understanding 3D spatial relationships remains a major limitation of current Vision-Language Models (VLMs). Prior work has addressed this issue by creating spatial question-answering (QA) datasets based on single images or indoor videos. However, real-world embodied AI agents such as robots and self-driving cars typically rely on ego-centric, multi-view observations. To this end, we introduce Ego3D-Bench, a new benchmark designed to evaluate the spatial reasoning abilities of VLMs using ego-centric, multi-view outdoor data. Ego3D-Bench comprises over 8,600 QA pairs, created with significant involvement from human annotators to ensure quality and diversity. We benchmark 16 SOTA VLMs, including GPT-4o, Gemini1.5-Pro, InternVL3, and Qwen2.5-VL. Our results reveal a notable performance gap between human level scores and VLM performance, highlighting that current VLMs still fall short of human level spatial understanding. To bridge this gap, we propose Ego3D-VLM, a post-training framework that enhances 3D spatial reasoning of VLMs. Ego3D-VLM generates cognitive map based on estimated global 3D coordinates, resulting in 12% average improvement on multi-choice QA and 56% average improvement on absolute distance estimation. Ego3D-VLM is modular and can be integrated with any existing VLM. Together, Ego3D-Bench and Ego3D-VLM offer valuable tools for advancing toward human level spatial understanding in real-world, multi-view environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
