arXiv 2406.136422025-03-19cs.CV

SpatialBot：基于视觉语言模型的精准空间理解

SpatialBot: Precise Spatial Understanding with Vision Language Models

School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Stanford University（斯坦福大学）；BAAI（北京智源人工智能研究院）；Peking University（北京大学）；University of Oxford（牛津大学）；Southeast University（东南大学）

Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, Bo Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉语言模型空间理解能力不足的问题，研究者提出SpatialBot模型，结合RGB与深度图像，并构建SpatialQA数据集与SpatialBench基准，实验证明其能显著提升相关性能。

摘要：

视觉语言模型（VLM）在二维图像理解领域已取得亮眼表现，但作为具身智能（Embodied AI）基础的空间理解能力仍存在短板。本文提出SpatialBot，通过同时输入RGB图像与深度图像提升空间理解效果。此外，我们构建了SpatialQA数据集，包含多级深度相关问题，用于训练VLM的深度理解能力。最后，我们提出SpatialBench基准，可全面评估VLM在不同层级的空间理解能力。在我们的空间理解基准、通用VLM基准及具身智能任务上开展的大量实验表明，经SpatialQA训练的SpatialBot性能提升显著。模型、代码与数据可在https://github.com/BAAI-DCAI/SpatialBot获取。

英文摘要：

Vision Language Models (VLMs) have achieved impressive performance in 2D image understanding, however they are still struggling with spatial understanding which is the foundation of Embodied AI. In this paper, we propose SpatialBot for better spatial understanding by feeding both RGB and depth images. Additionally, we have constructed the SpatialQA dataset, which involves multi-level depth-related questions to train VLMs for depth understanding. Finally, we present SpatialBench to comprehensively evaluate VLMs' capabilities in spatial understanding at different levels. Extensive experiments on our spatial-understanding benchmark, general VLM benchmarks and Embodied AI tasks, demonstrate the remarkable improvements of SpatialBot trained on SpatialQA. The model, code and data are available at https://github.com/BAAI-DCAI/SpatialBot.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。