MoniRefer: 一种基于道路基础设施的现实世界大规模多模态数据集用于3D视觉定位
MoniRefer: A Real-world Large-scale Multi-modal Dataset based on Roadside Infrastructure for 3D Visual Grounding
AI总结：
本文提出MoniRefer数据集和Moni3DVG方法，用于解决户外监控场景下的3D视觉定位问题。
摘要：
3D视觉定位旨在定位在3D点云场景中与给定自然语言句子语义对应的物体。这对于道路基础设施系统在复杂交通环境中解释自然语言并定位相关目标物体至关重要。然而，大多数现有的3D视觉定位数据集和方法集中在室内和户外驾驶场景，由于道路上方基础设施传感器捕获的点云-文本数据稀缺，户外监控场景仍未被探索。在本文中，我们引入了一个新的任务：针对户外监控场景的3D视觉定位，这使交通场景的基础设施级理解超越了以自身车辆视角。为了支持这一任务，我们构建了MoniRefer，这是首个现实世界的大规模多模态数据集，用于道路级的3D视觉定位。该数据集包含约136,018个对象和411,128个自然语言表达，这些数据来自多个复杂的交通交叉口的现实环境。为了确保数据集的质量和准确性，我们手动验证了所有语言描述和3D标签。此外，我们还提出了一种新的端到端方法，称为Moni3DVG，该方法利用图像和点云提供的丰富外观信息和几何光学信息进行多模态特征学习和3D物体定位。对所提出基准的广泛实验和消融研究证明了我们方法的优越性和有效性。我们的数据集和代码将被发布。
英文摘要：
3D visual grounding aims to localize the object in 3D point cloud scenes that semantically corresponds to given natural language sentences. It is very critical for roadside infrastructure system to interpret natural languages and localize relevant target objects in complex traffic environments. However, most existing datasets and approaches for 3D visual grounding focus on the indoor and outdoor driving scenes, outdoor monitoring scenarios remain unexplored due to scarcity of paired point cloud-text data captured by roadside infrastructure sensors. In this paper, we introduce a novel task of 3D Visual Grounding for Outdoor Monitoring Scenarios, which enables infrastructure-level understanding of traffic scenes beyond the ego-vehicle perspective. To support this task, we construct MoniRefer, the first real-world large-scale multi-modal dataset for roadside-level 3D visual grounding. The dataset consists of about 136,018 objects with 411,128 natural language expressions collected from multiple complex traffic intersections in the real-world environments. To ensure the quality and accuracy of the dataset, we manually verified all linguistic descriptions and 3D labels for objects. Additionally, we also propose a new end-to-end method, named Moni3DVG, which utilizes the rich appearance information provided by images and geometry and optical information from point cloud for multi-modal feature learning and 3D object localization. Extensive experiments and ablation studies on the proposed benchmarks demonstrate the superiority and effectiveness of our method. Our dataset and code will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
