SpatialRGPT：视觉语言模型中的接地空间推理
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
AI总结：
本文提出SpatialRGPT框架，通过3D场景图数据筛选管道和深度信息插件模块增强视觉语言模型的空间感知与推理能力，并发布SpatialRGPT-Bench基准，显著提升了空间推理性能且具备强泛化能力。
摘要：
视觉语言模型（VLMs）在2D视觉和语言任务中表现出色，但其推理空间布局的能力仍然有限。在本研究中，我们提出了Spatial Region GPT（SpatialRGPT）以增强VLM的空间感知与推理能力。SpatialRGPT通过两项关键创新提升了VLM的空间理解：(1) 一个数据筛选管道，能够从3D场景图中有效学习区域表示；(2) 一个灵活的插件模块，用于将深度信息整合到现有VLM的视觉编码器中。在推理阶段，当提供用户指定的区域提议时，SpatialRGPT能够准确感知其相对方向和距离。此外，我们提出了SpatialRGPT-Bench，这是一个包含室内、室外和模拟环境真实3D标注的基准，用于评估VLM的3D空间认知。结果表明，SpatialRGPT显著提升了空间推理任务的性能，无论是否带有局部区域提示。该模型还展现出强大的泛化能力，能有效推理复杂空间关系，并作为机器人任务的区域感知密集奖励标注器。代码、数据集和基准已发布于https://www.anjiecheng.me/SpatialRGPT。
英文摘要：
Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spatial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. SpatialRGPT advances VLMs' spatial understanding through two key innovations: (1) a data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (2) a flexible plugin module for integrating depth information into the visual encoder of existing VLMs. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialRGBT-Bench, a benchmark with ground-truth 3D annotations encompassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in VLMs. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as a region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at https://www.anjiecheng.me/SpatialRGPT
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
