Lowis3D：语言驱动的开放世界实例级3D场景理解
Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding
AI总结：
提出Lowis3D框架，利用预训练VL模型生成多视图描述建立3D与语义关联，并设计层次化点-描述关联和去偏实例定位，在3D分割任务上显著超越基线。
摘要：
开放世界实例级场景理解旨在定位并识别标注数据集中未出现的未见物体类别。该任务具有挑战性，因为模型既需要定位新颖的3D物体，又需要推断其语义类别。近期2D开放世界感知取得进展的关键因素在于互联网上大规模图文对的可用性，这些图文对涵盖了广泛的词汇概念。然而，由于3D-文本对的稀缺性，这种成功难以在3D场景中复制。为了应对这一挑战，我们提出利用预训练的视觉-语言（VL）基础模型，这些模型从图文对中编码了丰富的知识，为3D场景的多视图图像生成描述。这使我们能够在3D形状与语义丰富的描述之间建立显式关联。此外，为了增强从描述中学习细粒度视觉-语义表示以进行物体级分类，我们设计了层次化点-描述关联方法，以学习利用3D点与多视图图像之间3D几何关系的语义感知嵌入。另外，为了解决开放世界环境下新类别的定位挑战，我们开发了去偏实例定位，该方法涉及在无标签数据上使用实例级伪监督训练物体分组模块。这显著提升了实例分组的泛化能力，从而提高了准确定位新颖物体的能力。我们在3D语义、实例和全景分割任务上进行了广泛实验，涵盖了三个数据集的室内和室外场景。我们的方法在语义分割（例如34.5%$\sim$65.3%）、实例分割（例如21.8%$\sim$54.0%）和全景分割（例如14.7%$\sim$43.3%）上以显著优势超越了基线方法。代码将会公开。
英文摘要：
Open-world instance-level scene understanding aims to locate and recognize unseen object categories that are not present in the annotated dataset. This task is challenging because the model needs to both localize novel 3D objects and infer their semantic categories. A key factor for the recent progress in 2D open-world perception is the availability of large-scale image-text pairs from the Internet, which cover a wide range of vocabulary concepts. However, this success is hard to replicate in 3D scenarios due to the scarcity of 3D-text pairs. To address this challenge, we propose to harness pre-trained vision-language (VL) foundation models that encode extensive knowledge from image-text pairs to generate captions for multi-view images of 3D scenes. This allows us to establish explicit associations between 3D shapes and semantic-rich captions. Moreover, to enhance the fine-grained visual-semantic representation learning from captions for object-level categorization, we design hierarchical point-caption association methods to learn semantic-aware embeddings that exploit the 3D geometry between 3D points and multi-view images. In addition, to tackle the localization challenge for novel classes in the open-world setting, we develop debiased instance localization, which involves training object grouping modules on unlabeled data using instance-level pseudo supervision. This significantly improves the generalization capabilities of instance grouping and thus the ability to accurately locate novel objects. We conduct extensive experiments on 3D semantic, instance, and panoptic segmentation tasks, covering indoor and outdoor scenes across three datasets. Our method outperforms baseline methods by a significant margin in semantic segmentation (e.g. 34.5%$\sim$65.3%), instance segmentation (e.g. 21.8%$\sim$54.0%) and panoptic segmentation (e.g. 14.7%$\sim$43.3%). Code will be available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
