具有隐式和显式几何的3D感知视觉语言模型
3D-Aware VLMs with Implicit and Explicit Geometries
浏览论文内容
中文总结 AI 辅助
研究针对多数2D视觉输入的VLM处理3D任务困难的问题，提出VLM-IE3D框架，通过引入隐式和显式几何令牌及3D感知适配器，融合几何表示与视觉线索，在多种3D任务中表现优异。
中文摘要 AI 辅助
尽管取得了快速进展，但大多数基于2D视觉输入构建的现有视觉语言模型（VLM）在处理需要细粒度空间理解和推理的各种3D任务时往往面临困难。为了弥合这一差距，我们提出了VLM-IE3D，这是一个统一框架，通过为VLM配备从RGB视频中学习的隐式和显式3D几何来增强其3D空间感知。我们的VLM-IE3D引入了隐式几何令牌（IGT）来从输入视频中捕获高级几何先验，以及补充的显式几何令牌（EGT）来编码来自重建3D属性的详细几何结构。在此基础上，VLM-IE3D还配备了一个3D感知适配器，可有效地将这两种几何表示与2D视觉线索融合。这种仅基于RGB的设计为细粒度空间理解和推理注入了强大的3D归纳偏差，而无需任何额外的3D输入。广泛的实验表明，VLM-IE3D在包括3D视频检测、3D视觉定位、3D密集字幕和空间推理在内的各种3D任务中始终取得优异性能。代码和模型可在该https URL获取。
英文摘要
Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.
发表机构
- Nanyang Technological University（南洋理工大学）
- DAMO Academy, Alibaba Group（阿里巴巴达摩院）
- HuPan Lab（湖畔实验室）
机构由 AI 辅助整理，请以论文原文为准。
