VGGDrive: 通过跨视角几何 grounding 为自动驾驶赋能 Vision-Language 模型
VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving
AI总结：
VGGDrive通过引入跨视角几何 grounding 机制，提升Vision-Language模型在自动驾驶任务中的性能表现。
摘要：
跨视角3D几何建模能力在自动驾驶中的重要性不言而喻，但现有的Vision-Language Models（VLMs）本质上缺乏这种能力，导致其表现一般。虽然一些有前途的方法试图通过构建问答数据进行辅助训练来缓解这一问题，但它们仍然未能从根本上为VLMs配备处理多种评估协议的能力。因此，我们开辟了一条新道路，主张将成熟的3D基础模型的跨视角几何 grounding 融入VLMs中，弥补自动驾驶中的这一关键能力缺口。在此精神下，我们提出了一种新的架构VGGDrive，通过跨视角几何 grounding 为自动驾驶赋能Vision-Language模型。具体而言，为了将冻结的视觉3D模型中的跨视角3D几何特征与VLM的2D视觉特征连接起来，我们引入了一种即插即用的跨视角3D几何赋能器（CVGE）。CVGE解耦了基础VLM架构，并通过分层自适应注入机制有效地使VLM获得3D特征。大量实验表明，VGGDrive在五个自动驾驶基准测试中提升了基础VLM的表现，包括跨视角风险感知、运动预测和轨迹规划等任务。我们认为成熟的3D基础模型可以通过有效的整合来赋能自动驾驶任务，并希望我们的初步探索能够向自动驾驶社区展示这种范式潜力。
英文摘要：
The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-Language Models (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-language models with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
