In recent years, reinforcement learning (RL)-based methods for learning driving policies have gained increasing attention in the autonomous driving community and have achieved remarkable progress in various driving scenarios. However, traditional RL approaches rely on manually engineered rewards, which require extensive human effort and often lack generalizability. To address these limitations, we propose \textbf{VLM-RL}, a unified framework that integrates pre-trained Vision-Language Models (VLMs) with RL to generate reward signals using image observation and natural language goals. The core of VLM-RL is the contrasting language goal (CLG)-as-reward paradigm, which uses positive and negative language goals to generate semantic rewards. We further introduce a hierarchical reward synthesis approach that combines CLG-based semantic rewards with vehicle state information, improving reward stability and offering a more comprehensive reward signal. Additionally, a batch-processing technique is employed to optimize computational efficiency during training. Extensive experiments in the CARLA simulator demonstrate that VLM-RL outperforms state-of-the-art baselines, achieving a 10.5\% reduction in collision rate, a 104.6\% increase in route completion rate, and robust generalization to unseen driving scenarios. Furthermore, VLM-RL can seamlessly integrate almost any standard RL algorithms, potentially revolutionizing the existing RL paradigm that relies on manual reward engineering and enabling continuous performance improvements. The demo video and code can be accessed at: https://zilin-huang.github.io/VLM-RL-website.
VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving
Lyles School of Civil and Construction Engineering, Purdue University(普渡大学莱尔斯土木与建筑工程学院);Department of Civil and Environmental Engineering, University of Wisconsin-Madison(威斯康星大学麦迪逊分校土木与环境工程系);Google(谷歌)
Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
University of California, Irvine(加利福尼亚大学欧文分校);Shanghai AI Laboratory(上海人工智能实验室);National University of Singapore(新加坡国立大学);S-Lab, Nanyang Technological University(南洋理工大学S实验室);The University of Hong Kong(香港大学)
Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Northwest A&F University(西北农林科技大学);Nanyang Technological University(南洋理工大学);Jinan University(济南大学)
LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
Department of Civil and Environmental Engineering, University of Michigan(土木与环境工程系,密歇根大学);University of Michigan Transportation Research Institute(密歇根大学交通研究所)
VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Li Auto Inc.(Li汽车公司);the School of Aeronautics and Astronautics, Xiamen University(厦门大学航空航天学院);The Hong Kong University of Science and Technology(香港科技大学)
V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models
organization= Department of Civil;Environmental Engineering, University of Wisconsin–Madison , city= Madison , state= WI , postcode= 53706 , country= USA;organization= College of Computing;Data Science, Nanyang Technological University , city= Singapore , postcode= 639798 , country= Singapore;organization= College of Transportation, Tongji University , city= Shanghai , postcode= 201804 , country= China;organization= Zachry Department of Civil;Environmental Engineering, Texas A\&M University , city= College Station , state= TX , postcode= 77840 , country= USA;organization= School of Civil;Environmental Engineering, Cornell University , city= Ithaca , state= NY , postcode= 14853 , country= USA
DepthVision: Enabling Robust Vision-Language Models with GAN-Based LiDAR-to-RGB Synthesis for Autonomous Driving
Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich(机器人学、人工智能与实时系统教授会,慕尼黑技术大学);Computer Science and Engineering Department, University of California Merced(计算机科学与工程系,加州大学默塞德分校)
VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving
Department of Civil and Environmental Engineering, The Pennsylvania State University(宾夕法尼亚州立大学土木与环境工程系);Department of Civil Engineering, Stony Brook University(石溪大学土木工程系)
VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));National University of Singapore(新加坡国立大学);The Hong Kong University of Science and Technology(香港科学与技术大学)
VLMs Guided Interpretable Decision Making for Autonomous Driving
Department of Computer Science, Tulane University(路易斯安那大学计算机科学系);Qualcomm(高通公司);Department of Industrial and Systems Engineering, North Carolina State University(北卡罗来纳州立大学工业与系统工程系)
SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models
West China Biomedical Big Data Center, West China Hospital, SCU(西昌生物医学大数据中心、西昌医院、SCU);NTU;USTC;TeleAI, China Telecom(TeleAI、中国电信);BUPT;Tsinghua University(清华大学);HKUST(Guangzhou)(HKUST(广州))