N3D-VLM：原生3D接地使视觉-语言模型在3D场景中实现精确的空间推理
N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models
AI总结：
N3D-VLM通过原生3D感知能力提升视觉-语言模型在3D场景中的空间推理精度与解释性。
摘要：
尽管当前多模态模型能够基于2D图像回答问题，但它们缺乏内在的3D物体感知能力，限制了它们对3D场景中空间关系和深度线索的理解能力。在本文中，我们提出了N3D-VLM，一种新的统一框架，无缝整合原生3D物体感知与3D感知的视觉推理，使模型能够实现精确的3D接地和可解释的空间理解。与传统端到端模型直接从RGB/RGB-D输入预测答案不同，我们的方法使模型具备原生3D物体感知能力，使其能够根据文本描述直接在3D空间中定位物体。在准确的3D物体定位基础上，模型进一步进行显式的3D推理，实现更可解释和结构化的空间理解。为支持这些能力的稳健训练，我们开发了一条可扩展的数据构建管道，利用深度估计将大规模2D注释提升到3D空间，显著增加了3D物体接地数据的多样性和覆盖范围，数据量超过现有最大单张图像3D检测数据集的六倍。此外，该管道生成了针对思维链（CoT）推理的3D空间问答数据集，促进了3D物体定位和3D空间推理的联合训练。实验结果表明，我们的统一框架不仅在3D接地任务上实现了最先进的性能，而且在视觉-语言模型的3D空间推理方面也 consistently 超过了现有方法。
英文摘要：
While current multimodal models can answer questions based on 2D images, they lack intrinsic 3D object perception, limiting their ability to comprehend spatial relationships and depth cues in 3D scenes. In this work, we propose N3D-VLM, a novel unified framework that seamlessly integrates native 3D object perception with 3D-aware visual reasoning, enabling both precise 3D grounding and interpretable spatial understanding. Unlike conventional end-to-end models that directly predict answers from RGB/RGB-D inputs, our approach equips the model with native 3D object perception capabilities, enabling it to directly localize objects in 3D space based on textual descriptions. Building upon accurate 3D object localization, the model further performs explicit reasoning in 3D, achieving more interpretable and structured spatial understanding. To support robust training for these capabilities, we develop a scalable data construction pipeline that leverages depth estimation to lift large-scale 2D annotations into 3D space, significantly increasing the diversity and coverage for 3D object grounding data, yielding over six times larger than the largest existing single-image 3D detection dataset. Moreover, the pipeline generates spatial question-answering datasets that target chain-of-thought (CoT) reasoning in 3D, facilitating joint training for both 3D object localization and 3D spatial reasoning. Experimental results demonstrate that our unified framework not only achieves state-of-the-art performance on 3D grounding tasks, but also consistently surpasses existing methods in 3D spatial reasoning in vision-language model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
