arXiv 2405.128212025-02-09cs.ROcs.CV

Talk2Radar：连接自然语言与4D毫米波雷达以实现3D指代表达理解

Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension

Institute of Deep Perception Technology, JITRI（JITRI深度感知技术研究所）；University of Liverpool（利物浦大学）；University of Southampton（南安普顿大学）；Momoni AI（Momoni人工智能公司）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Thrust of Artificial Intelligence, HKUST (GZ)（香港科技大学（广州）人工智能学域）；Thrust of Intelligent Transportation, HKUST (GZ)（香港科技大学（广州）智能交通学域）

Runwei Guan, Ruixiao Zhang, Ningwei Ouyang, Jianan Liu, Ka Lok Man, Xiaohao Cai, Ming Xu, Jeremy Smith, Eng Gee Lim, Yutao Yue, Hui Xiong

备注：: Accepted by ICRA 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文构建了首个面向4D毫米波雷达3D指代表达理解的数据集Talk2Radar，并提出T-RadarNet模型，通过Deformable-FPN和Gated Graph Fusion实现高效点云建模与跨模态融合，取得SOTA性能。

摘要：

具身感知对于智能车辆和机器人在交互式环境理解中至关重要。然而，这些进展主要集中在视觉方面，对使用3D建模传感器的关注有限，限制了在响应包含定性和定量查询的提示时对物体的全面理解。近年来，作为一种成本可负担且具有前景的车载传感器，4D毫米波雷达提供了比传统雷达更密集的点云，并能感知物体的语义和物理特征，从而增强感知系统的可靠性。为了促进雷达场景中自然语言驱动的上下文理解在3D视觉定位中的发展，我们构建了首个数据集Talk2Radar，该数据集为3D指代表达理解（REC）桥接了这两种模态。Talk2Radar包含8,682个指代提示样本，涉及20,558个被指代物体。此外，我们提出了一种新颖的模型T-RadarNet，用于点云上的3D REC，在Talk2Radar数据集上相比同类方法取得了最先进（SOTA）的性能。我们精心设计了Deformable-FPN和Gated Graph Fusion，分别用于高效的点云特征建模以及雷达与文本特征之间的跨模态融合。全面的实验为基于雷达的3D REC提供了深入见解。我们在https://github.com/GuanRunwei/Talk2Radar发布了本项目。

英文摘要：

Embodied perception is essential for intelligent vehicles and robots in interactive environmental understanding. However, these advancements primarily focus on vision, with limited attention given to using 3D modeling sensors, restricting a comprehensive understanding of objects in response to prompts containing qualitative and quantitative queries. Recently, as a promising automotive sensor with affordable cost, 4D millimeter-wave radars provide denser point clouds than conventional radars and perceive both semantic and physical characteristics of objects, thereby enhancing the reliability of perception systems. To foster the development of natural language-driven context understanding in radar scenes for 3D visual grounding, we construct the first dataset, Talk2Radar, which bridges these two modalities for 3D Referring Expression Comprehension (REC). Talk2Radar contains 8,682 referring prompt samples with 20,558 referred objects. Moreover, we propose a novel model, T-RadarNet, for 3D REC on point clouds, achieving State-Of-The-Art (SOTA) performance on the Talk2Radar dataset compared to counterparts. Deformable-FPN and Gated Graph Fusion are meticulously designed for efficient point cloud feature modeling and cross-modal fusion between radar and text features, respectively. Comprehensive experiments provide deep insights into radar-based 3D REC. We release our project at https://github.com/GuanRunwei/Talk2Radar.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。