无需3D数据训练开放词汇单目3D目标检测模型
Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data
AI总结：
本文提出OVM3D-Det框架，仅用RGB图像和伪LiDAR自动标注，结合自适应侵蚀与LLM先验细化边界框，实现低成本开放词汇单目3D目标检测，在室内外场景中优于基线方法。
摘要：
开放词汇3D目标检测因其在自动驾驶和机器人领域的广泛应用而近来受到大量关注，其目标是在未见过的领域中有效识别新类别。然而，现有的基于点云的开放词汇3D检测模型受限于高昂的部署成本。本文提出一种新颖的开放词汇单目3D目标检测框架，名为OVM3D-Det，该框架仅使用RGB图像训练检测器，既具有成本效益，又可扩展到公开可用的数据。与传统方法不同，OVM3D-Det在输入或生成3D边界框时均不需要高精度LiDAR或3D传感器数据。取而代之的是，它利用开放词汇2D模型和伪LiDAR自动标注RGB图像中的3D目标，从而促进开放词汇单目3D检测器的学习。然而，直接使用由伪LiDAR导出的标签训练3D模型并不充分，因为从噪声点云中估计的边界框不精确，且严重遮挡的目标也难以处理。为解决这些问题，我们引入了两项创新设计：自适应伪LiDAR侵蚀和借助大语言模型先验知识的边界框细化。这些技术有效校准了3D标签，并实现了仅用RGB训练3D检测器。大量实验表明，OVM3D-Det在室内和室外场景中均优于基线方法。代码将予以发布。
英文摘要：
Open-vocabulary 3D object detection has recently attracted considerable attention due to its broad applications in autonomous driving and robotics, which aims to effectively recognize novel classes in previously unseen domains. However, existing point cloud-based open-vocabulary 3D detection models are limited by their high deployment costs. In this work, we propose a novel open-vocabulary monocular 3D object detection framework, dubbed OVM3D-Det, which trains detectors using only RGB images, making it both cost-effective and scalable to publicly available data. Unlike traditional methods, OVM3D-Det does not require high-precision LiDAR or 3D sensor data for either input or generating 3D bounding boxes. Instead, it employs open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in RGB images, fostering the learning of open-vocabulary monocular 3D detectors. However, training 3D models with labels directly derived from pseudo-LiDAR is inadequate due to imprecise boxes estimated from noisy point clouds and severely occluded objects. To address these issues, we introduce two innovative designs: adaptive pseudo-LiDAR erosion and bounding box refinement with prior knowledge from large language models. These techniques effectively calibrate the 3D labels and enable RGB-only training for 3D detectors. Extensive experiments demonstrate the superiority of OVM3D-Det over baselines in both indoor and outdoor scenarios. The code will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
