ImOV3D：仅从2D图像学习开放词汇点云3D目标检测
ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images
AI总结：
本文提出ImOV3D框架，通过单目深度估计和渲染实现2D图像与3D点云间的灵活模态转换，利用伪多模态表示弥合模态鸿沟，在无真实3D训练数据或仅少量微调下显著提升开放词汇3D目标检测性能。
摘要：
开放词汇3D目标检测（OV-3Det）旨在泛化到训练阶段标注的有限基础类别之外。最大的瓶颈是标注3D数据的稀缺，而2D图像数据集则非常丰富且标注详尽。因此，利用2D图像中丰富的标注来缓解OV-3Det固有的数据稀缺性是直观的做法。在本文中，我们通过探索仅使用2D图像来学习OV-3Det的潜力，将任务设置推向极限。这种设置的主要挑战在于训练图像与测试点云之间的模态鸿沟，这阻碍了2D知识有效整合到OV-3Det中。为了应对这一挑战，我们提出了一种新颖的框架ImOV3D，利用包含图像和点云（PC）的伪多模态表示来弥合模态鸿沟。ImOV3D的关键在于灵活的模态转换，其中2D图像可以通过单目深度估计提升为3D，也可以通过渲染从3D场景中导出。这使得训练图像和测试点云能够统一为共同的图像-PC表示，既包含丰富的2D语义信息，又融合了3D空间数据的深度和结构特征。我们谨慎地进行这种转换，以最小化训练和测试用例之间的域差距。在两个基准数据集SUNRGBD和ScanNet上的大量实验表明，即使在没有真实3D训练数据的情况下，ImOV3D也显著优于现有方法。在加入少量真实3D数据进行微调后，其性能也显著超越了先前的最先进水平。代码和预训练模型已发布于https://github.com/yangtiming/ImOV3D。
英文摘要：
Open-vocabulary 3D object detection (OV-3Det) aims to generalize beyond the limited number of base categories labeled during the training phase. The biggest bottleneck is the scarcity of annotated 3D data, whereas 2D image datasets are abundant and richly annotated. Consequently, it is intuitive to leverage the wealth of annotations in 2D images to alleviate the inherent data scarcity in OV-3Det. In this paper, we push the task setup to its limits by exploring the potential of using solely 2D images to learn OV-3Det. The major challenges for this setup is the modality gap between training images and testing point clouds, which prevents effective integration of 2D knowledge into OV-3Det. To address this challenge, we propose a novel framework ImOV3D to leverage pseudo multimodal representation containing both images and point clouds (PC) to close the modality gap. The key of ImOV3D lies in flexible modality conversion where 2D images can be lifted into 3D using monocular depth estimation and can also be derived from 3D scenes through rendering. This allows unifying both training images and testing point clouds into a common image-PC representation, encompassing a wealth of 2D semantic information and also incorporating the depth and structural characteristics of 3D spatial data. We carefully conduct such conversion to minimize the domain gap between training and test cases. Extensive experiments on two benchmark datasets, SUNRGBD and ScanNet, show that ImOV3D significantly outperforms existing methods, even in the absence of ground truth 3D training data. With the inclusion of a minimal amount of real 3D data for fine-tuning, the performance also significantly surpasses previous state-of-the-art. Codes and pre-trained models are released on the https://github.com/yangtiming/ImOV3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
