G$^2$VLM：基于几何的视觉语言模型，实现统一的3D重建与空间推理
G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
AI总结：
G$^2$VLM通过统一3D重建与空间推理，提升视觉语言模型在空间智能任务中的性能。
摘要：
视觉语言模型（VLMs）在空间智能方面仍然缺乏鲁棒性，在空间理解和推理任务上表现不佳。我们归因于缺乏能够从2D图像中重建3D空间的视觉几何学习过程。我们提出了G$^2$VLM，一种基于几何的视觉语言模型，连接了空间智能的两个基本方面：空间3D重建和空间理解。G$^2$VLM原生利用学习的3D视觉几何特征，通过上下文学习和交错推理直接预测3D属性并增强空间推理任务。我们的统一设计在空间理解方面具有高度可扩展性：它在大量多视角图像和视频数据上进行训练，同时利用通常只能从难以收集的注释中获得的3D视觉先验优势。实验结果表明，G$^2$VLM在两项任务中都表现出色，其在3D重建任务上的表现与最先进的前馈3D重建模型相当，在空间理解和推理任务上也取得了更好或更具竞争力的结果。通过将语义强的VLM与低层次的3D视觉任务统一起来，我们希望G$^2$VLM能够为社区提供一个强大的基准，并解锁更多未来应用，如3D场景编辑。
英文摘要：
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
