Occ-VLM: 面向室内场景理解的占用接地视觉语言模型
Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding
AI总结：
提出Occ-VLM，仅用姿态RGB图像和单一2D视觉编码器，通过重建3D占用作为几何先验，实现统一的3D场景理解，在占用预测、3D VQA和密集描述任务上达到领先水平。
摘要：
近期，视觉语言模型（VLM）在3D场景理解方面取得了显著进展，推动了具身智能和机器人视觉等应用的发展。然而，现有方法通常要么直接依赖显式的3D输入（如点云或RGB-D序列），要么引入额外的3D几何编码器从2D图像中推导出3D感知的视觉标记。这种设计在结构上将3D几何感知与通过视觉语言预训练学到的丰富2D语义解耦，阻碍了统一3D视觉语言表示的发展。在这项工作中，我们提出了Occ-VLM，一个仅基于姿态RGB图像并采用单一2D视觉编码器的3D场景理解新框架。具体而言，Occ-VLM重建3D场景占用作为辅助几何先验，用于将前景2D标记与3D空间进行空间关联。然后，这些标记由大型语言模型（LLM）解码，实现统一的场景理解。大量实验表明，Occ-VLM实现了准确的几何感知和稳健的视觉语言推理：在多视角占用预测上达到最先进性能，同时在3D视觉问答（VQA）和3D密集描述基准上与使用3D输入的VLM表现相当。
英文摘要：
Recently, vision-language models (VLMs) have made significant progress in 3D scene understanding, driving advances in applications such as embodied intelligence and robotic vision. However, existing approaches typically either rely directly on explicit 3D inputs (e.g., point clouds or RGB-D sequences), or introduce an additional 3D geometry encoder to derive 3D-aware visual tokens from 2D images. Such designs structurally decouple 3D geometric perception from the rich 2D semantics learned via vision-language pre-training, hindering the development of a unified 3D vision-language representation. In this work, we propose Occ-VLM, a novel framework for 3D scene understanding that operates purely on posed RGB images and employs a single 2D vision encoder. Specifically, Occ-VLM reconstructs 3D scene occupancy as an auxiliary geometric prior, which is utilized to spatially associate foreground 2D tokens with 3D space. These tokens are then decoded by a Large Language Model (LLM) for unified scene understanding. Extensive experiments demonstrate that Occ-VLM achieves both accurate geometric perception and robust vision-language reasoning: it attains state-of-the-art performance on multi-view occupancy prediction, while performing on par with 3D-input VLMs on 3D Visual Question Answering (VQA) and 3D dense captioning benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
