EmbodiedSplat: 在线前馈语义3DGS用于开放词汇3D场景理解
EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding
AI总结：
EmbodiedSplat通过在线前馈3DGS实现开放词汇3D场景理解，结合CLIP编码和3D U-Net提升语义重建效率与泛化能力。
摘要：
对3D场景的即时理解对于具身任务至关重要，其中智能体必须在线且几乎实时地构建和理解3D场景。在本研究中，我们提出了EmbodiedSplat，一种在线前馈3DGS用于开放词汇场景理解，能够同时从流媒体图像中进行在线3D重建和3D语义理解。与现有开放词汇3DGS方法通常局限于离线或每场景优化设置不同，我们的目标是双重的：1）从超过300张流媒体图像中在线重建整个场景的语义嵌入3DGS。2）通过前馈设计实现对新场景的高泛化能力，并在结合实时2D模型时支持接近实时的3D语义重建。为实现这些目标，我们提出了一个带有CLIP全局代码本的在线稀疏系数场，其中将2D CLIP嵌入绑定到每个3D高斯体，同时最小化内存消耗并保留CLIP的完整语义泛化能力。此外，我们通过3D U-Net聚合3DGS的不完整点云，生成3D几何感知的CLIP特征，以补偿3D几何先验到2D导向语言嵌入。在ScanNet、ScanNet++和Replica等多样化的室内数据集上进行了广泛的实验，证明了我们方法的有效性和效率。查看我们的项目页面：https://0nandon.github.io/EmbodiedSplat/。
英文摘要：
Understanding a 3D scene immediately with its exploration is essential for embodied tasks, where an agent must construct and comprehend the 3D scene in an online and nearly real-time manner. In this study, we propose EmbodiedSplat, an online feed-forward 3DGS for open-vocabulary scene understanding that enables simultaneous online 3D reconstruction and 3D semantic understanding from the streaming images. Unlike existing open-vocabulary 3DGS methods which are typically restricted to either offline or per-scene optimization setting, our objectives are two-fold: 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner. 2) Highly generalizable to novel scenes with feed-forward design and supports nearly real-time 3D semantic reconstruction when combined with real-time 2D models. To achieve these objectives, we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian while minimizing memory consumption and preserving the full semantic generalizability of CLIP. Furthermore, we generate 3D geometric-aware CLIP features by aggregating the partial point cloud of 3DGS through 3D U-Net to compensate the 3D geometric prior to 2D-oriented language embeddings. Extensive experiments on diverse indoor datasets, including ScanNet, ScanNet++, and Replica, demonstrate both the effectiveness and efficiency of our method. Check out our project page in https://0nandon.github.io/EmbodiedSplat/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
