具有快速渲染和预训练视觉-语言对齐的通用化机器人3D视觉-语言模型
Generalized Robot 3D Vision-Language Model with Fast Rendering and Pre-Training Vision-Language Alignment
AI总结：
本文提出WS3D++框架，通过层级特征对齐预训练、知识蒸馏和区域级语义对比学习，在标签稀缺条件下提升3D点云开放词汇分割与检测性能。
摘要：
深度神经网络模型在闭集设定下并使用完整标签进行训练时，已经在3D场景理解方面取得了显著进展。然而，主要瓶颈在于，在各种真实世界应用中，这些模型无法识别训练类别之外的任何未见新类别。因此，我们迫切需要一个能够同时适用于3D点云分割和检测的框架，尤其是在标签相当稀缺的情况下。本工作提出了一个通用且直接的框架，用于在带标注场景非常有限时处理3D场景理解。为了从预训练视觉-语言模型中提取新类别知识，我们提出了一种层级化特征对齐预训练和知识蒸馏策略，以从大规模视觉-语言模型中提取并蒸馏有意义的信息，从而助力开放词汇场景理解任务。为了促进潜在实例判别并保证效率，我们针对点云提出了无监督区域级语义对比学习方案，利用神经网络的高置信度预测在多个阶段判别中间特征嵌入。在有限重建情形下，我们提出的名为WS3D++的方法在大规模ScanNet基准的语义分割和实例分割两项任务上均排名第1。在室内和室外场景上的大量实验证明了本方法在数据高效学习和开放世界少样本学习两方面的有效性。代码已公开发布于：https://drive.google.com/drive/folders/1M58V-PtR8DBEwD296zJkNg_m2qq-MTAP?usp=sharing。
英文摘要：
Deep neural network models have achieved remarkable progress in 3D scene understanding while trained in the closed-set setting and with full labels. However, the major bottleneck is that these models do not have the capacity to recognize any unseen novel classes beyond the training categories in diverse real-world applications. Therefore, we are in urgent need of a framework that can simultaneously be applicable to both 3D point cloud segmentation and detection, particularly in the circumstances where the labels are rather scarce. This work presents a generalized and straightforward framework for dealing with 3D scene understanding when the labeled scenes are quite limited. To extract knowledge for novel categories from the pre-trained vision-language models, we propose a hierarchical feature-aligned pre-training and knowledge distillation strategy to extract and distill meaningful information from large-scale vision-language models, which helps benefit the open-vocabulary scene understanding tasks. To encourage latent instance discrimination and to guarantee efficiency, we propose the unsupervised region-level semantic contrastive learning scheme for point clouds, using confident predictions of the neural network to discriminate the intermediate feature embeddings at multiple stages. In the limited reconstruction case, our proposed approach, termed WS3D++, ranks 1st on the large-scale ScanNet benchmark on both the task of semantic segmentation and instance segmentation. Extensive experiments with both indoor and outdoor scenes demonstrated the effectiveness of our approach in both data-efficient learning and open-world few-shot learning. The code is made publicly available at: https://drive.google.com/drive/folders/1M58V-PtR8DBEwD296zJkNg_m2qq-MTAP?usp=sharing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
