V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models
organization= Department of Civil;Environmental Engineering, University of Wisconsin–Madison , city= Madison , state= WI , postcode= 53706 , country= USA;organization= College of Computing;Data Science, Nanyang Technological University , city= Singapore , postcode= 639798 , country= Singapore;organization= College of Transportation, Tongji University , city= Shanghai , postcode= 201804 , country= China;organization= Zachry Department of Civil;Environmental Engineering, Texas A\&M University , city= College Station , state= TX , postcode= 77840 , country= USA;organization= School of Civil;Environmental Engineering, Cornell University , city= Ithaca , state= NY , postcode= 14853 , country= USA
Junwei You, Haotian Shi, Zhuoyu Jiang, Zilin Huang, Rui Gan, Keshu Wu, Xi Cheng, Xiaopeng Li, Bin Ran
Vehicle-to-everything (V2X) cooperation has emerged as a promising paradigm to overcome the perception limitations of classical autonomous driving by leveraging information from both ego-vehicle and infrastructure sensors. However, effectively fusing heterogeneous visual and semantic information while ensuring robust trajectory planning remains a significant challenge. This paper introduces V2X-VLM, a novel end-to-end (E2E) cooperative autonomous driving framework based on vision-language models (VLMs). V2X-VLM integrates multiperspective camera views from vehicles and infrastructure with text-based scene descriptions to enable a more comprehensive understanding of driving environments. Specifically, we propose a contrastive learning-based mechanism to reinforce the alignment of heterogeneous visual and textual characteristics, which enhances the semantic understanding of complex driving scenarios, and employ a knowledge distillation strategy to stabilize training. Experiments on a large real-world dataset demonstrate that V2X-VLM achieves state-of-the-art trajectory planning accuracy, significantly reducing L2 error and collision rate compared to existing cooperative autonomous driving baselines. Ablation studies validate the contributions of each component. Moreover, the evaluation of robustness and efficiency highlights the practicality of V2X-VLM for real-world deployment to enhance overall autonomous driving safety and decision-making.
VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving
Lyles School of Civil and Construction Engineering, Purdue University(普渡大学莱尔斯土木与建筑工程学院);Department of Civil and Environmental Engineering, University of Wisconsin-Madison(威斯康星大学麦迪逊分校土木与环境工程系);Google(谷歌)
VLM-C4L: Continual Core Dataset Learning with Corner Case Optimization via Vision-Language Models for Autonomous Driving
City University of Hong Kong(香港城市大学);Soochow University(苏州大学);McGill University(麦吉尔大学);Hon Hai Research Institute(鸿海研究院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
Department of Civil and Environmental Engineering, University of Michigan(土木与环境工程系,密歇根大学);University of Michigan Transportation Research Institute(密歇根大学交通研究所)
Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
University of California, Irvine(加利福尼亚大学欧文分校);Shanghai AI Laboratory(上海人工智能实验室);National University of Singapore(新加坡国立大学);S-Lab, Nanyang Technological University(南洋理工大学S实验室);The University of Hong Kong(香港大学)
VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));National University of Singapore(新加坡国立大学);The Hong Kong University of Science and Technology(香港科学与技术大学)
Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Northwest A&F University(西北农林科技大学);Nanyang Technological University(南洋理工大学);Jinan University(济南大学)
AppleVLM: End-to-end Autonomous Driving with Advanced Perception and Planning-Enhanced Vision-Language Models
Shenzhen Key Lab for Advanced Motion Control and Modern Automation Equipments(深圳先进运动控制系统与现代自动化装备重点实验室);Guangdong Provincial Key Laboratory of Intelligent Morphing Mechanisms and Adaptive Robotics(广东省智能变形机制与适应机器人省重点实验室);School of Intelligence Science and Engineering(智能科学与工程学院);Harbin Institute of Technology(哈尔滨工业大学);National Key Laboratory of Smart Farm Technologies and Systems(国家智能农业技术与系统重点实验室);Autonomous Driving Center(自动驾驶中心);Shanghai Utopilot Technology Co.Ltd.(上海驭目科技有限公司)
COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learning
School of Mechanical and Aerospace Engineering, Nanyang Technological University(南洋理工大学机械与航空航天工程学院);National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人机混合增强智能国家重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究院)
VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving
Department of Civil and Environmental Engineering, The Pennsylvania State University(宾夕法尼亚州立大学土木与环境工程系);Department of Civil Engineering, Stony Brook University(石溪大学土木工程系)
CoLMIN: LLM-based Multi-Decision Path Negotiation for Cooperative Autonomous Driving
School of future Transportation, Chang’an University(长安大学未来交通学院);Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);School of information, Renmin University of China(中国人民大学信息学院);School of Artificial Intelligence and Robotics, Hunan University(湖南大学人工智能与机器人学院)
Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作 Medianet 创新中心);Shanghai AI Laboratory(上海人工智能实验室);The University of Warwick(华威大学)
VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Li Auto Inc.(Li汽车公司);the School of Aeronautics and Astronautics, Xiamen University(厦门大学航空航天学院);The Hong Kong University of Science and Technology(香港科技大学)
V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
Department of Civil and Environmental Engineering, University of Wisconsin–Madison(威斯康星大学麦迪逊分校土木与环境工程系);Department of Civil and Architectural Engineering and Construction Management, University of Wyoming(怀俄明大学土木与建筑工程及施工管理系);School of Transportation, Southeast University(东南大学交通学院)
Department of Civil and Environmental Engineering, University of Wisconsin–Madison(威斯康星大学麦迪逊分校土木与环境工程系);College of Computing and Data Science, Nanyang Technological University(南洋理工大学 computing and Data Science 学院);College of Transportation, Tongji University(同济大学交通运输学院)