PanopticRecon：利用开放词汇实例分割实现零样本全景重建
PanopticRecon: Leverage Open-vocabulary Instance Segmentation for Zero-shot Panoptic Reconstruction
AI总结：
本文提出PanopticRecon，利用开放词汇实例分割和三维实例图分割实现零样本全景重建，在ScanNet V2和KITTI-360上取得领先效果。
摘要：
全景重建是三维场景理解中一项具有挑战性的任务。然而，现有大多数方法严重依赖预训练的语义分割模型和已知的三维物体边界框来进行三维全景分割，这对于自然场景并不适用。本文提出了一种新颖的基于场景RGB-D图像的零样本全景重建方法。在零样本分割方面，我们利用开放词汇实例分割，但必须面对部分标注和实例关联的挑战。我们通过借助稠密广义特征传播部分标签，并构建三维实例图来关联二维实例ID，从而应对这两项挑战。具体而言，我们利用部分标签为广义语义特征学习一个分类器，为具有稠密蒸馏特征的场景提供完整标签。此外，我们将实例关联形式化为三维实例图分割问题，使我们能够充分利用场景几何先验和所有二维实例掩码来推断全局唯一的伪三维实例ID。我们的方法在室内数据集ScanNet V2和室外数据集KITTI-360上优于最先进的方法，证明了我们的图分割方法和重建网络的有效性。
英文摘要：
Panoptic reconstruction is a challenging task in 3D scene understanding. However, most existing methods heavily rely on pre-trained semantic segmentation models and known 3D object bounding boxes for 3D panoptic segmentation, which is not available for in-the-wild scenes. In this paper, we propose a novel zero-shot panoptic reconstruction method from RGB-D images of scenes. For zero-shot segmentation, we leverage open-vocabulary instance segmentation, but it has to face partial labeling and instance association challenges. We tackle both challenges by propagating partial labels with the aid of dense generalized features and building a 3D instance graph for associating 2D instance IDs. Specifically, we exploit partial labels to learn a classifier for generalized semantic features to provide complete labels for scenes with dense distilled features. Moreover, we formulate instance association as a 3D instance graph segmentation problem, allowing us to fully utilize the scene geometry prior and all 2D instance masks to infer global unique pseudo 3D instance ID. Our method outperforms state-of-the-art methods on the indoor dataset ScanNet V2 and the outdoor dataset KITTI-360, demonstrating the effectiveness of our graph segmentation method and reconstruction network.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
