arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21186cs.CV
收藏

Spa3R:用于3D视觉推理的预测空间场建模

Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning

Huazhong University of Science & Technology(华中科技大学);Horizon Robotics;D-Robotics Intern at D-Robotics(D-Robotics 实习生)

Haoyi Jiang, Liu Liu, Xinjie Wang, Yonghao He, Wei Sui, Zhizhong Su, Wenyu Liu, Xinggang Wang

AI总结:

Spa3R通过自监督学习从多视角图像中提取统一的空间表示,提升3D视觉推理能力,实现与VLMs的结合,达到3D VQA任务的高准确率。

摘要:

尽管视觉-语言模型(VLMs)在2D视觉理解方面表现出色,但其对3D空间的理解和推理能力——空间智能的核心——仍然较为肤浅。当前的方法试图弥合这一领域差距,要么依赖于显式的3D模态,要么通过向VLMs添加部分、视图条件的几何先验进行增强。然而,这些方法限制了可扩展性,并最终使语言模型承担了从稀疏线索中隐式重建整体3D几何的不恰当任务。在本文中,我们主张空间智能可以源自2D视觉本身,而不是通过显式的空间指令微调来施加。为此,我们引入了Spa3R,一种自监督框架,它直接从未经处理的多视角图像中学习统一的、视图不变的空间表示。Spa3R基于所提出的预测空间场建模(PSFM)范式,其中Spa3R学习合成任意未见过的视图的特征场,条件于一个紧凑的潜在表示,从而内部化对底层3D场景的完整且一致的理解。我们进一步将预训练的Spa3R编码器整合到现有VLMs中,通过轻量级适配器形成Spa3-VLM,从而将语言推理扎根于全球空间上下文中。在具有挑战性的VSI-Bench实验中,Spa3-VLM在3D VQA任务上实现了58.6%的最先进准确率,显著优于先前方法。这些结果突显了PSFM作为推动空间智能发展的可扩展路径。代码可在https://github.com/hustvl/Spa3R上获得。

英文摘要:

Vision-language models excel at 2D visual understanding but remain limited in 3D spatial reasoning. Existing approaches either depend on explicit 3D modalities, which limits scalability, or inject partial, view-conditioned geometric priors and leave the language model to recover global scene structure from sparse cues. We introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation from unposed multi-view RGB images. Its Predictive Spatial Field Modeling objective compresses context views into a compact latent representation and predicts aligned geometric and semantic feature fields at novel viewpoints, thereby encouraging coherent encoding of scene geometry and layout. We integrate the pre-trained Spa3R Encoder into a vision-language model through a lightweight residual cross-attention adapter, yielding Spa3-VLM and grounding language reasoning in global spatial context. Spa3-VLM achieves an average score of 58.6% on VSI-Bench and delivers leading or competitive performance across three additional spatial reasoning benchmarks. These results demonstrate that predictive spatial representation learning provides an effective visual foundation for 3D reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑