NuGrounding：自动驾驶中的多视角三维视觉定位框架
NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Driving
AI总结：
本文提出NuGrounding，首个自动驾驶多视角三维视觉定位基准，通过层次化指令构建及融合多模态大模型与检测模型的方法，显著提升定位精度与召回率。
摘要：
多视角三维视觉定位对于自动驾驶车辆理解自然语言并在复杂环境中定位目标物体至关重要。然而，现有数据集和方法存在语言指令粒度粗糙、三维几何推理与语言理解整合不足的问题。为此，我们引入了NuGrounding，这是自动驾驶领域首个大规模多视角三维视觉定位基准。我们提出了一种层次化定位（HoG）方法构建NuGrounding，以生成分层多级指令，确保全面覆盖人类指令模式。为应对这一具有挑战性的数据集，我们提出了一种新颖范式，将多模态大语言模型（MLLMs）的指令理解能力与专业检测模型的精确定位能力无缝结合。我们的方法引入了两个解耦的任务标记和一个上下文查询，以聚合三维几何信息和语义指令，随后通过融合解码器细化空间-语义特征融合以实现精确定位。大量实验表明，我们的方法显著优于从代表性三维场景理解方法改编的基线，精度达到0.59，召回率达到0.64，分别提升了50.8%和54.7%。
英文摘要：
Multi-view 3D visual grounding is critical for autonomous driving vehicles to interpret natural languages and localize target objects in complex environments. However, existing datasets and methods suffer from coarse-grained language instructions, and inadequate integration of 3D geometric reasoning with linguistic comprehension. To this end, we introduce NuGrounding, the first large-scale benchmark for multi-view 3D visual grounding in autonomous driving. We present a Hierarchy of Grounding (HoG) method to construct NuGrounding to generate hierarchical multi-level instructions, ensuring comprehensive coverage of human instruction patterns. To tackle this challenging dataset, we propose a novel paradigm that seamlessly combines instruction comprehension abilities of multi-modal LLMs (MLLMs) with precise localization abilities of specialist detection models. Our approach introduces two decoupled task tokens and a context query to aggregate 3D geometric information and semantic instructions, followed by a fusion decoder to refine spatial-semantic feature fusion for precise localization. Extensive experiments demonstrate that our method significantly outperforms the baselines adapted from representative 3D scene understanding methods by a significant margin and achieves 0.59 in precision and 0.64 in recall, with improvements of 50.8% and 54.7%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
