arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.07977cs.CV
收藏

Gaga: 通过3D感知记忆库分组任意高斯体

Gaga: Group Any Gaussians via 3D-aware Memory Bank

University of California, Merced(加州大学默塞德分校);NVIDIA Research(英伟达研究);Google DeepMind(谷歌DeepMind);Atmanity Inc.(Atmanity公司)

Weijie Lyu, Xueting Li, Abhijit Kundu, Yi-Hsuan Tsai, Ming-Hsuan Yang

AI总结:

提出Gaga框架,利用零样本类别无关分割模型预测的不一致2D掩码,通过3D感知记忆库关联不同视角下的物体掩码,实现开放世界3D场景的重建与分割。

摘要:

我们介绍了Gaga,一个通过利用零样本类别无关分割模型预测的不一致2D掩码来重建和分割开放世界3D场景的框架。与先前依赖视频对象跟踪或对比学习方法的3D场景分割方法不同,Gaga利用空间信息并通过新颖的3D感知记忆库有效关联不同相机姿态下的物体掩码。通过消除训练图像中连续视角变化的假设,Gaga展现出对相机姿态变化的鲁棒性,尤其有利于稀疏采样图像,确保精确的掩码标签一致性。此外,Gaga可兼容来自不同来源的2D分割掩码,并与不同的开放世界零样本类别无关分割模型展现出稳健性能,显著增强了其通用性。大量的定性和定量评估表明,Gaga的性能优于现有最先进方法,凸显了其在3D场景理解与操作等实际应用中的潜力。

英文摘要:

We introduce Gaga, a framework that reconstructs and segments open-world 3D scenes by leveraging inconsistent 2D masks predicted by zero-shot class-agnostic segmentation models. Contrasted to prior 3D scene segmentation approaches that rely on video object tracking or contrastive learning methods, Gaga utilizes spatial information and effectively associates object masks across diverse camera poses through a novel 3D-aware memory bank. By eliminating the assumption of continuous view changes in training images, Gaga demonstrates robustness to variations in camera poses, particularly beneficial for sparsely sampled images, ensuring precise mask label consistency. Furthermore, Gaga accommodates 2D segmentation masks from diverse sources and demonstrates robust performance with different open-world zero-shot class-agnostic segmentation models, significantly enhancing its versatility. Extensive qualitative and quantitative evaluations demonstrate that Gaga performs favorably against state-of-the-art methods, emphasizing its potential for real-world applications such as 3D scene understanding and manipulation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑