生成模型了解空间：解锁隐式3D先验用于场景理解
Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
AI总结：
本文提出通过大规模视频生成模型的隐式空间先验提升场景理解，引入VEGA-3D框架，利用预训练视频扩散模型作为潜在世界模拟器，通过时空特征提取与语义融合增强大语言模型的几何信息，实验验证其在3D场景理解、空间推理和具身操控中的优越性。
摘要：
尽管多模态大语言模型展现了出色的语义能力，但它们常常面临空间盲区，难以进行细粒度几何推理和物理动态。现有解决方案通常依赖显式的3D模态或复杂的几何架构，受限于数据稀缺性和泛化挑战。在本文中，我们提出了一种范式转变，通过利用大规模视频生成模型中的隐式空间先验。我们假设为了合成时间一致的视频，这些模型本质上学习了稳健的3D结构先验和物理定律。我们引入VEGA-3D（Video Extracted Generative Awareness），一个即插即用的框架，通过重新利用预训练的视频扩散模型作为潜在世界模拟器。通过从中间噪声水平提取时空特征，并通过令牌级自适应门控融合机制将其与语义表示结合，我们增强了大语言模型的密集几何提示，而无需显式的3D监督。在3D场景理解、空间推理和具身操控基准上的广泛实验表明，我们的方法优于最先进的基线，验证了生成先验为物理世界理解提供了可扩展的基础。代码可在https://github.com/H-EmbodVis/VEGA-3D上公开获取。
英文摘要：
While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale video generation models. We posit that to synthesize temporally coherent videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a token-level adaptive gated fusion mechanism, we enrich MLLMs with dense geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial reasoning, and embodied manipulation benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
