VLM-Auto：面向复杂道路场景、具备类人行为与理解能力的基于 VLM 的自动驾驶助手
VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenes
AI总结：
提出 VLM-Auto 自动驾驶助手，利用 VLM 理解复杂道路场景并生成可调节驾驶行为，通过 CARLA 与 ROS2 在单张 4090 上验证，并发布微调数据集。
摘要：
近期关于将大型语言模型用于自动驾驶的研究在规划与控制方面展现出前景。然而，高昂的计算需求和幻觉问题仍对准确的轨迹预测与控制信号生成构成挑战。确定性算法具备可靠性，但缺乏对复杂驾驶场景的适应能力，并且难以处理上下文和不确定性。为解决这一问题，我们提出 VLM-Auto，这是一种新型自动驾驶助手系统，可使自动驾驶车辆基于对道路场景的理解获得可调节的驾驶行为。文中提出了一条涉及 CARLA 仿真器和 Robot Operating System 2（ROS2）的流程，用于验证系统有效性；该流程仅使用单块 Nvidia 4090 24G GPU，同时利用视觉语言模型（VLM）的文本输出能力。此外，我们还贡献了一个数据集，其中包含图像集及相应提示集，用于微调系统的 VLM 模块。在 CARLA 实验中，我们的系统在数据集的 5 类标签上取得了 97.82% 的平均精度。在真实世界驾驶数据集上，系统在夜间场景和阴沉场景中取得了 96.97% 的预测准确率。我们的 VLM-Auto 数据集将发布于 https://github.com/ZionGo6/VLM-Auto。
英文摘要：
Recent research on Large Language Models for autonomous driving shows promise in planning and control. However, high computational demands and hallucinations still challenge accurate trajectory prediction and control signal generation. Deterministic algorithms offer reliability but lack adaptability to complex driving scenarios and struggle with context and uncertainty. To address this problem, we propose VLM-Auto, a novel autonomous driving assistant system to empower the autonomous vehicles with adjustable driving behaviors based on the understanding of road scenes. A pipeline involving the CARLA simulator and Robot Operating System 2 (ROS2) verifying the effectiveness of our system is presented, utilizing a single Nvidia 4090 24G GPU while exploiting the capacity of textual output of the Visual Language Model (VLM). Besides, we also contribute a dataset containing an image set and a corresponding prompt set for fine-tuning the VLM module of our system. In CARLA experiments, our system achieved $97.82\%$ average precision on 5 types of labels in our dataset. In the real-world driving dataset, our system achieved $96.97\%$ prediction accuracy in night scenes and gloomy scenes. Our VLM-Auto dataset will be released at https://github.com/ZionGo6/VLM-Auto.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
