EgoSplat：利用语言嵌入的3D Gaussian Splatting进行开放词汇第一人称场景理解
EgoSplat: Open-Vocabulary Egocentric Scene Understanding with Language Embedded 3D Gaussian Splatting
AI总结：
本文提出EgoSplat框架，结合SAM2多视图实例特征聚合与时空瞬态预测，在开放词汇第一人称场景理解的定位和分割任务上取得新SOTA。
摘要：
与典型场景理解任务相比，第一人称场景表现出频繁遮挡、多样视角和动态交互。遮挡和多样视角会导致多视图语义不一致，而动态对象可能充当短暂干扰物，给语义特征建模引入伪影。为应对这些挑战，我们提出EgoSplat，这是一个用于开放词汇第一人称场景理解的语言嵌入3D Gaussian Splatting框架。该框架设计了多视图一致的实例特征聚合方法，利用SAM2的分割和跟踪能力，为每个实例选择性地聚合跨视图的互补特征，确保场景语义表示的精确性。此外，构建了实例感知的时空瞬态预测模块，通过融合多视图实例之间的时空关联，提升预测中的空间完整性和时间连续性，有效减少第一人称场景语义重建中的伪影。EgoSplat在两个数据集的定位和分割任务上均达到最先进性能；在ADT数据集上，其定位精度较现有方法提升8.2%，分割mIoU提升3.7%，并为开放词汇第一人称场景理解设立了新基准。代码将公开发布。
英文摘要：
Egocentric scenes exhibit frequent occlusions, varied viewpoints, and dynamic interactions compared to typical scene understanding tasks. Occlusions and varied viewpoints can lead to multi-view semantic inconsistencies, while dynamic objects may act as transient distractors, introducing artifacts into semantic feature modeling. To address these challenges, we propose EgoSplat, a language-embedded 3D Gaussian Splatting framework for open-vocabulary egocentric scene understanding. A multi-view consistent instance feature aggregation method is designed to leverage the segmentation and tracking capabilities of SAM2 to selectively aggregate complementary features across views for each instance, ensuring precise semantic representation of scenes. Additionally, an instance-aware spatial-temporal transient prediction module is constructed to improve spatial integrity and temporal continuity in predictions by incorporating spatial-temporal associations across multi-view instances, effectively reducing artifacts in the semantic reconstruction of egocentric scenes. EgoSplat achieves state-of-the-art performance in both localization and segmentation tasks on two datasets, outperforming existing methods with a 8.2% improvement in localization accuracy and a 3.7% improvement in segmentation mIoU on the ADT dataset, and setting a new benchmark in open-vocabulary egocentric scene understanding. The code will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
