arXiv 2411.106392025-03-10cs.CVcs.AIcs.CLcs.LG

MTA：面向BEV感知与描述的多模态任务对齐

MTA: Multimodal Task Alignment for BEV Perception and Captioning

Bosch Research North America（博世北美研究院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；Purdue University（普渡大学）

Yunsheng Ma, Burhaneddin Yaman, Xin Ye, Jingru Luo, Feng Tao, Abhirup Mallik, Ziran Wang, Liu Ren

备注：: 10 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MTA多模态任务对齐框架，通过BEV-语言对齐和检测-描述对齐两大机制，在不增加推理计算开销的情况下同时提升BEV感知与描述性能，并在nuScenes和TOD3Cap上显著超越现有基线。

摘要：

基于鸟瞰图（BEV）的3D感知在自动驾驶应用中起着至关重要的作用。大语言模型的兴起激发了人们对基于BEV的描述任务的兴趣，以理解周围环境中的物体行为。然而，现有方法将感知和描述视为独立任务，只关注单一任务的性能，忽视了多模态对齐的潜在益处。为了弥合模态之间的这一差距，我们提出了MTA，一种新颖的多模态任务对齐框架，可同时提升BEV感知和描述性能。MTA包含两个关键组件：（1）BEV-语言对齐（BLA），一种将BEV场景表示与真值语言表示对齐的上下文学习机制；（2）检测-描述对齐（DCA），一种将检测输出与描述输出对齐的跨模态提示机制。MTA在训练过程中无缝集成到最先进的基线模型中，在运行时不会增加额外的计算复杂度。在nuScenes和TOD3Cap数据集上的大量实验表明，MTA在两项任务上均显著优于最先进的基线模型，在具有挑战性的稀有感知场景中实现了10.7%的提升，在描述任务中实现了9.2%的提升。这些结果凸显了统一对齐在协调基于BEV的感知与描述方面的有效性。

英文摘要：

Bird's eye view (BEV)-based 3D perception plays a crucial role in autonomous driving applications. The rise of large language models has spurred interest in BEV-based captioning to understand object behavior in the surrounding environment. However, existing approaches treat perception and captioning as separate tasks, focusing on the performance of only one task and overlooking the potential benefits of multimodal alignment. To bridge this gap between modalities, we introduce MTA, a novel multimodal task alignment framework that boosts both BEV perception and captioning. MTA consists of two key components: (1) BEV-Language Alignment (BLA), a contextual learning mechanism that aligns the BEV scene representations with ground-truth language representations, and (2) Detection-Captioning Alignment (DCA), a cross-modal prompting mechanism that aligns detection and captioning outputs. MTA seamlessly integrates into state-of-the-art baselines during training, adding no extra computational complexity at runtime. Extensive experiments on the nuScenes and TOD3Cap datasets show that MTA significantly outperforms state-of-the-art baselines in both tasks, achieving a 10.7% improvement in challenging rare perception scenarios and a 9.2% improvement in captioning. These results underscore the effectiveness of unified alignment in reconciling BEV-based perception and captioning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。