arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.15544cs.ROcs.AIcs.CV
收藏

VLM-RL:一个用于安全自动驾驶的统一视觉语言模型与强化学习框架

VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Purdue University(普渡大学)

Zilin Huang, Zihao Sheng, Yansong Qu, Junwei You, Sikai Chen

AI总结:

本文提出VLM-RL框架,将预训练视觉语言模型与强化学习结合,通过对比语言目标生成语义奖励并融合车辆状态信息,有效减少了对人工奖励设计的依赖,在自动驾驶任务中显著降低了碰撞率并提升了路线完成率。

摘要:

近年来,基于强化学习(RL)的驾驶策略学习方法在自动驾驶领域受到越来越多的关注,并在各种驾驶场景中取得了显著进展。然而,传统的RL方法依赖人工设计的奖励,这需要大量人力且通常缺乏泛化能力。为了解决这些局限性,我们提出了VLM-RL,这是一个将预训练的视觉语言模型(VLM)与RL相集成的统一框架,利用图像观测和自然语言目标来生成奖励信号。VLM-RL的核心是对比语言目标(CLG)作为奖励的范式,它使用正面和负面的语言目标来生成语义奖励。我们进一步引入了分层奖励合成方法,将基于CLG的语义奖励与车辆状态信息相结合,提高了奖励稳定性并提供了更全面的奖励信号。此外,采用批处理技术来优化训练过程中的计算效率。在CARLA模拟器中的大量实验表明,VLM-RL优于最先进的基线方法,碰撞率降低了10.5%,路线完成率提高了104.6%,并且对未见过的驾驶场景具有强大的泛化能力。此外,VLM-RL可以无缝集成几乎任何标准RL算法,有望彻底改变依赖手动奖励工程的现有RL范式,并实现持续的性能提升。演示视频和代码可在以下网址获取:https://zilin-huang.github.io/VLM-RL-website。

英文摘要:

In recent years, reinforcement learning (RL)-based methods for learning driving policies have gained increasing attention in the autonomous driving community and have achieved remarkable progress in various driving scenarios. However, traditional RL approaches rely on manually engineered rewards, which require extensive human effort and often lack generalizability. To address these limitations, we propose \textbf{VLM-RL}, a unified framework that integrates pre-trained Vision-Language Models (VLMs) with RL to generate reward signals using image observation and natural language goals. The core of VLM-RL is the contrasting language goal (CLG)-as-reward paradigm, which uses positive and negative language goals to generate semantic rewards. We further introduce a hierarchical reward synthesis approach that combines CLG-based semantic rewards with vehicle state information, improving reward stability and offering a more comprehensive reward signal. Additionally, a batch-processing technique is employed to optimize computational efficiency during training. Extensive experiments in the CARLA simulator demonstrate that VLM-RL outperforms state-of-the-art baselines, achieving a 10.5\% reduction in collision rate, a 104.6\% increase in route completion rate, and robust generalization to unseen driving scenarios. Furthermore, VLM-RL can seamlessly integrate almost any standard RL algorithms, potentially revolutionizing the existing RL paradigm that relies on manual reward engineering and enabling continuous performance improvements. The demo video and code can be accessed at: https://zilin-huang.github.io/VLM-RL-website.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑