4DLangVGGT: 4D语言-视觉几何 grounded Transformer
4DLangVGGT: 4D Language-Visual Geometry Grounded Transformer
AI总结：
4DLangVGGT是一种基于Transformer的统一框架，用于4D语言 grounding，通过整合几何感知和语言对齐，提升4D场景理解的泛化能力和部署效率。
摘要：
构建4D语言场对于具身AI、增强/虚拟现实以及4D场景理解至关重要，因为它们为动态环境提供了丰富的语义表示，并能够在复杂场景中实现开放词汇查询。然而，现有方法在构建4D语义场时主要依赖于场景特定的高斯散射，这需要每个场景的优化，具有有限的泛化能力，并且难以扩展到现实应用。为了解决这些限制，我们提出了4DLangVGGT，这是首个基于Transformer的前馈统一框架，用于4D语言 grounding，它在单一架构中联合整合了几何感知和语言对齐。4DLangVGGT有两个关键组件：4D视觉几何Transformer，StreamVGGT，它捕捉动态场景的时空几何表示；以及语义桥接解码器（SBD），它将具有几何意识的特征投影到语言对齐的语义空间，从而在增强语义可解释性的同时保持结构保真度。与以往依赖于昂贵的每场景优化的方法不同，4DLangVGGT可以在多个动态场景上联合训练，并在推理时直接应用，实现了部署效率和强大泛化能力。这种设计显著提高了大规模部署的实用性，并为开放词汇4D场景理解建立了新的范式。在HyperNeRF和Neu3D数据集上的实验表明，我们的方法不仅泛化效果良好，而且实现了最先进的性能，在单场景训练下达到2%的提升，在多场景训练下达到1%的改进。我们的代码已发布在https://github.com/hustvl/4DLangVGGT。
英文摘要：
Constructing 4D language fields is crucial for embodied AI, augmented/virtual reality, and 4D scene understanding, as they provide enriched semantic representations of dynamic environments and enable open-vocabulary querying in complex scenarios. However, existing approaches to 4D semantic field construction primarily rely on scene-specific Gaussian splatting, which requires per-scene optimization, exhibits limited generalization, and is difficult to scale to real-world applications. To address these limitations, we propose 4DLangVGGT, the first Transformer-based feed-forward unified framework for 4D language grounding, that jointly integrates geometric perception and language alignment within a single architecture. 4DLangVGGT has two key components: the 4D Visual Geometry Transformer, StreamVGGT, which captures spatio-temporal geometric representations of dynamic scenes; and the Semantic Bridging Decoder (SBD), which projects geometry-aware features into a language-aligned semantic space, thereby enhancing semantic interpretability while preserving structural fidelity. Unlike prior methods that depend on costly per-scene optimization, 4DLangVGGT can be jointly trained across multiple dynamic scenes and directly applied during inference, achieving both deployment efficiency and strong generalization. This design significantly improves the practicality of large-scale deployment and establishes a new paradigm for open-vocabulary 4D scene understanding. Experiments on HyperNeRF and Neu3D datasets demonstrate that our approach not only generalizes effectively but also achieves state-of-the-art performance, achieving up to 2% gains under per-scene training and 1% improvements under multi-scene training. Our code released in https://github.com/hustvl/4DLangVGGT
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
