arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.18002cs.CVcs.AIcs.CL
收藏

Loc3R-VLM:基于语言的定位与3D推理的视觉语言模型

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

Microsoft Spatial AI Lab(微软空间AI实验室);ETH Zurich(苏黎世联邦理工学院);EPFL(苏黎世联邦理工学院)

Kevin Qu, Haozhe Qi, Mihai Dusmanu, Mahdi Rad, Rui Wang, Marc Pollefeys

AI总结:

Loc3R-VLM通过结合全局布局重建和显式情境建模,提升视觉语言模型的3D空间理解能力,在语言定位和3D问答任务中取得最优性能。

摘要:

多模态大语言模型(MLLMs)在连接视觉和语言方面取得了显著进展,但仍在空间理解和视角感知推理方面面临挑战。本文提出Loc3R-VLM框架,通过单目视频输入为2D视觉语言模型赋予先进的3D理解能力。受人类空间认知启发,Loc3R-VLM采用两个联合目标:全局布局重建以构建场景结构的整体表示,以及显式情境建模以锚定以自身为中心的视角。这些目标提供了直接的空间监督,使感知和语言在3D上下文中扎根。为确保几何一致性和度量尺度对齐,我们利用从预训练3D基础模型中提取的轻量级相机姿态先验。Loc3R-VLM在基于语言的定位任务中取得最佳性能,并在场景化和通用3D问答基准上优于现有2D和视频方法,证明了我们的空间监督框架能够实现强大的3D理解。项目页面:https://kevinqu7.github.io/loc3r-vlm

英文摘要:

Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑