SD-VLM：基于深度编码视觉语言模型的空间测量与理解
SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
AI总结：
本文提出SD-VLM框架，通过构建包含70万问答对的大规模空间测量数据集MSMU和引入深度位置编码方法，显著提升了视觉语言模型的三维空间定量测量与理解能力，并在多个基准上超越GPT-4o等现有模型。
摘要：
尽管视觉语言模型（VLM）在二维语义视觉理解方面表现出色，但由于二维图像空间表示能力的不足，其对三维空间关系进行定量推理的能力仍未得到充分探索。本文分析了阻碍VLM空间理解能力的问题，并提出SD-VLM，一个通过两项关键贡献显著增强VLM基础空间感知能力的新框架：（1）提出具有精确空间标注的大规模空间测量与理解（MSMU）数据集；（2）引入一种简单的深度位置编码方法以增强VLM的空间感知能力。MSMU数据集涵盖大规模定量空间任务，包含700K问答对、250万个物理数值标注以及10K个思维链增强样本。我们训练了SD-VLM，一个强大的通用型VLM，展现出卓越的定量空间测量与理解能力。SD-VLM不仅在我们提出的MSMU-Bench上取得了最先进的性能，还在其他空间理解基准（包括Q-Spatial和SpatialRGPT-Bench）上展现出空间泛化能力。大量实验表明，SD-VLM在MSMU-Bench上分别比GPT-4o和Intern-VL3-78B高出26.91%和25.56%。代码和模型已发布于https://github.com/cpystan/SD-VLM。
英文摘要：
While vision language models (VLMs) excel in 2D semantic visual understanding, their ability to quantitatively reason about 3D spatial relationships remains under-explored, due to the deficiency of 2D images' spatial representation ability. In this paper, we analyze the problem hindering VLMs' spatial understanding abilities and propose SD-VLM, a novel framework that significantly enhances fundamental spatial perception abilities of VLMs through two key contributions: (1) propose Massive Spatial Measuring and Understanding (MSMU) dataset with precise spatial annotations, and (2) introduce a simple depth positional encoding method strengthening VLMs' spatial awareness. MSMU dataset covers massive quantitative spatial tasks with 700K QA pairs, 2.5M physical numerical annotations, and 10K chain-of-thought augmented samples. We have trained SD-VLM, a strong generalist VLM which shows superior quantitative spatial measuring and understanding capability. SD-VLM not only achieves state-of-the-art performance on our proposed MSMU-Bench, but also shows spatial generalization abilities on other spatial understanding benchmarks including Q-Spatial and SpatialRGPT-Bench. Extensive experiments demonstrate that SD-VLM outperforms GPT-4o and Intern-VL3-78B by 26.91% and 25.56% respectively on MSMU-Bench. Code and models are released at https://github.com/cpystan/SD-VLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
