arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09251cs.ROcs.AIcs.LG
收藏

V2X-VLM:通过大型视觉-语言模型实现端到端车路协同自动驾驶

V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models

organization= Department of Civil;Environmental Engineering, University of Wisconsin–Madison , city= Madison , state= WI , postcode= 53706 , country= USA;organization= College of Computing;Data Science, Nanyang Technological University , city= Singapore , postcode= 639798 , country= Singapore;organization= College of Transportation, Tongji University , city= Shanghai , postcode= 201804 , country= China;organization= Zachry Department of Civil;Environmental Engineering, Texas A\&M University , city= College Station , state= TX , postcode= 77840 , country= USA;organization= School of Civil;Environmental Engineering, Cornell University , city= Ithaca , state= NY , postcode= 14853 , country= USA

Junwei You, Haotian Shi, Zhuoyu Jiang, Zilin Huang, Rui Gan, Keshu Wu, Xi Cheng, Xiaopeng Li, Bin Ran

AI总结:

本文提出V2X-VLM,一种基于视觉-语言模型的端到端车路协同自动驾驶框架,通过融合多视角视觉与文本描述,利用对比学习和知识蒸馏,显著降低轨迹规划误差和碰撞率,提升驾驶安全。

摘要:

车联万物(V2X)合作已成为一种有前景的范式,通过利用自车和基础设施传感器的信息来克服传统自动驾驶的感知局限性。然而,在确保鲁棒轨迹规划的同时,有效融合异构的视觉和语义信息仍然是一个重大挑战。本文介绍了V2X-VLM,一种基于视觉-语言模型(VLMs)的新型端到端(E2E)协同自动驾驶框架。V2X-VLM将来自车辆和基础设施的多视角摄像头视图与基于文本的场景描述相结合,以实现对驾驶环境更全面的理解。具体而言,我们提出了一种基于对比学习的机制,以加强异构视觉和文本特征的对齐,从而增强对复杂驾驶场景的语义理解,并采用知识蒸馏策略来稳定训练。在大型真实世界数据集上的实验表明,V2X-VLM实现了最先进的轨迹规划精度,与现有协同自动驾驶基线相比,显著降低了L2误差和碰撞率。消融研究验证了每个组件的贡献。此外,鲁棒性和效率的评估凸显了V2X-VLM在实际部署中的实用性,以增强整体自动驾驶安全性和决策制定。

英文摘要:

Vehicle-to-everything (V2X) cooperation has emerged as a promising paradigm to overcome the perception limitations of classical autonomous driving by leveraging information from both ego-vehicle and infrastructure sensors. However, effectively fusing heterogeneous visual and semantic information while ensuring robust trajectory planning remains a significant challenge. This paper introduces V2X-VLM, a novel end-to-end (E2E) cooperative autonomous driving framework based on vision-language models (VLMs). V2X-VLM integrates multiperspective camera views from vehicles and infrastructure with text-based scene descriptions to enable a more comprehensive understanding of driving environments. Specifically, we propose a contrastive learning-based mechanism to reinforce the alignment of heterogeneous visual and textual characteristics, which enhances the semantic understanding of complex driving scenarios, and employ a knowledge distillation strategy to stabilize training. Experiments on a large real-world dataset demonstrate that V2X-VLM achieves state-of-the-art trajectory planning accuracy, significantly reducing L2 error and collision rate compared to existing cooperative autonomous driving baselines. Ablation studies validate the contributions of each component. Moreover, the evaluation of robustness and efficiency highlights the practicality of V2X-VLM for real-world deployment to enhance overall autonomous driving safety and decision-making.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑