开放词汇SAM3D：迈向免训练开放词汇3D场景理解
Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding
AI总结：
本文提出免训练的OV-SAM3D框架，结合SAM分割与RAM开放标签，无需场景先验知识即可实现开放词汇3D场景理解，并在ScanNet200和nuScenes上超越现有方法。
摘要：
开放词汇3D场景理解在该领域提出了重大挑战。近期工作试图将视觉-语言模型中嵌入的知识从2D迁移到3D领域。然而，这些方法通常需要来自特定3D场景数据集的先验知识，限制了它们在开放世界场景中的适用性。Segment Anything Model（SAM）已展现出卓越的零样本分割能力，促使我们研究其在无需训练的情况下理解3D场景的潜力。在本文中，我们提出OV-SAM3D，一种免训练方法，包含一个用于理解开放词汇3D场景的通用框架。该框架旨在对任意3D场景执行理解任务，无需场景的先验知识。具体而言，我们的方法由两个关键子模块组成：首先，我们通过生成超点作为初始3D提示来启动流程，并使用从SAM导出的分割掩码细化这些提示。此外，我们随后将专门设计的重叠分数表与来自Recognize Anything Model（RAM）的开放标签相结合，以生成带有开放世界标签的最终3D实例。在ScanNet200和nuScenes数据集上的实证评估表明，我们的方法在未知的开放世界环境中超越了现有的开放词汇方法。
英文摘要：
Open-vocabulary 3D scene understanding presents a significant challenge in the field. Recent works have sought to transfer knowledge embedded in vision-language models from 2D to 3D domains. However, these approaches often require prior knowledge from specific 3D scene datasets, limiting their applicability in open-world scenarios. The Segment Anything Model (SAM) has demonstrated remarkable zero-shot segmentation capabilities, prompting us to investigate its potential for comprehending 3D scenes without training. In this paper, we introduce OV-SAM3D, a training-free method that contains a universal framework for understanding open-vocabulary 3D scenes. This framework is designed to perform understanding tasks for any 3D scene without requiring prior knowledge of the scene. Specifically, our method is composed of two key sub-modules: First, we initiate the process by generating superpoints as the initial 3D prompts and refine these prompts using segment masks derived from SAM. Moreover, we then integrate a specially designed overlapping score table with open tags from the Recognize Anything Model (RAM) to produce final 3D instances with open-world labels. Empirical evaluations on the ScanNet200 and nuScenes datasets demonstrate that our approach surpasses existing open-vocabulary methods in unknown open-world environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
