利用2D-3D视觉-语言蒸馏的3D开放词汇全景分割
3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation
AI总结：
针对3D全景分割难以识别未知类别的问题，提出融合可学习LiDAR特征与冻结CLIP特征并进行对象级、体素级蒸馏的开放词汇方法，在nuScenes和SemanticKITTI上显著超过强基线。
摘要：
3D全景分割是一项具有挑战性的感知任务，尤其是在自动驾驶中。它旨在为场景中的3D点预测语义标注和实例标注。尽管先前的3D全景分割方法已在闭集基准上取得出色性能，但将这些方法泛化到未见过的物体类别和未见过的杂物类别仍是一个开放问题。对于未见过的物体类别，2D开放词汇分割仅依赖冻结的CLIP骨干网络并集成多个分类输出，已经取得了有前景的结果。然而，我们发现，由于逐掩膜分类质量较差，尤其是对于新的杂物类别，简单地将这些2D模型扩展到3D并不能保证良好性能。本文提出首个解决3D开放词汇全景分割的方法。我们的模型利用可学习LiDAR特征与密集冻结视觉CLIP特征之间的融合，使用单个分类头同时对基类和新类进行预测。为进一步提升新类上的分类性能并利用CLIP模型，我们提出两种新的损失函数：对象级蒸馏损失和体素级蒸馏损失。我们在nuScenes和SemanticKITTI数据集上的实验表明，本方法大幅优于强基线。
英文摘要：
3D panoptic segmentation is a challenging perception task, especially in autonomous driving. It aims to predict both semantic and instance annotations for 3D points in a scene. Although prior 3D panoptic segmentation approaches have achieved great performance on closed-set benchmarks, generalizing these approaches to unseen things and unseen stuff categories remains an open problem. For unseen object categories, 2D open-vocabulary segmentation has achieved promising results that solely rely on frozen CLIP backbones and ensembling multiple classification outputs. However, we find that simply extending these 2D models to 3D does not guarantee good performance due to poor per-mask classification quality, especially for novel stuff categories. In this paper, we propose the first method to tackle 3D open-vocabulary panoptic segmentation. Our model takes advantage of the fusion between learnable LiDAR features and dense frozen vision CLIP features, using a single classification head to make predictions for both base and novel classes. To further improve the classification performance on novel classes and leverage the CLIP model, we propose two novel loss functions: object-level distillation loss and voxel-level distillation loss. Our experiments on the nuScenes and SemanticKITTI datasets show that our method outperforms the strong baseline by a large margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
