全景视觉-语言特征场
Panoptic Vision-Language Feature Fields
AI总结：
本文提出 PVLFF，通过蒸馏2D视觉-语言特征并结合实例对比学习构建3D特征场，首次实现3D开放词汇全景分割，并在多个数据集上达到接近闭集SOTA的性能。
摘要：
最近，已有研究提出用于3D开放词汇语义分割的方法。这类方法能够根据运行时提供的文本描述，将场景分割为任意类别。在本文中，据我们所知，我们提出了首个用于3D场景开放词汇全景分割的算法。我们的算法 Panoptic Vision-Language Feature Fields（PVLFF）通过从预训练的2D模型中蒸馏视觉-语言特征来学习场景的语义特征场，并利用输入帧上的2D实例分割，通过对比学习联合拟合一个实例特征场。尽管未在目标类别上进行训练，我们的方法在 HyperSim、ScanNet 和 Replica 数据集上取得了与最先进的闭集3D系统相近的全景分割性能，并且在语义分割方面优于当前的3D开放词汇系统。我们对方法各组件进行了消融实验，以证明模型架构的有效性。我们的代码将在 https://github.com/ethz-asl/pvlff 提供。
英文摘要：
Recently, methods have been proposed for 3D open-vocabulary semantic segmentation. Such methods are able to segment scenes into arbitrary classes based on text descriptions provided during runtime. In this paper, we propose to the best of our knowledge the first algorithm for open-vocabulary panoptic segmentation in 3D scenes. Our algorithm, Panoptic Vision-Language Feature Fields (PVLFF), learns a semantic feature field of the scene by distilling vision-language features from a pretrained 2D model, and jointly fits an instance feature field through contrastive learning using 2D instance segments on input frames. Despite not being trained on the target classes, our method achieves panoptic segmentation performance similar to the state-of-the-art closed-set 3D systems on the HyperSim, ScanNet and Replica dataset and additionally outperforms current 3D open-vocabulary systems in terms of semantic segmentation. We ablate the components of our method to demonstrate the effectiveness of our model architecture. Our code will be available at https://github.com/ethz-asl/pvlff.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
