arXiv 2501.040032025-01-07cs.CVcs.RO

视觉语言模型（VLM）已为自动驾驶做好准备了吗？一项来自可靠性、数据与指标视角的实证研究

Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives

University of California, Irvine（加利福尼亚大学欧文分校）；Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）；The University of Hong Kong（香港大学）

Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, Liang Pan

备注：: Preprint; 41 pages, 32 figures, 16 tables; Project Page at https://drive-bench.github.io/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文构建涵盖17种设置、19,200帧和12种VLM的DriveBench，揭示VLM驾驶解释常依赖文本与常识而非视觉接地，并提出改进指标与可靠性提升路线。

摘要：

视觉语言模型（VLM）的最新进展激发了人们将其用于自动驾驶的兴趣，尤其是通过自然语言生成可解释的驾驶决策。然而，关于VLM天然能够为驾驶提供视觉接地、可靠且可解释的说明这一假设，在很大程度上仍未得到检验。为弥补这一空白，我们推出DriveBench，这是一个旨在评估VLM在17种设置（干净、损坏和纯文本输入）下可靠性的基准数据集，包含19,200帧、20,498个问答对、三种问题类型、四项主流驾驶任务，共计12种流行VLM。我们的发现表明，VLM往往会生成看似合理的回答，这些回答源自通用知识或文本线索，而非真正的视觉接地，在视觉输入退化或缺失时尤其如此。这种行为被数据集不平衡和不充分的评估指标所掩盖，在自动驾驶等安全关键场景中构成重大风险。我们进一步观察到，VLM在多模态推理方面存在困难，并对输入损坏表现出高度敏感性，从而导致性能不一致。为应对这些挑战，我们提出了改进的评估指标，优先考虑稳健的视觉接地和多模态理解。此外，我们强调利用VLM对损坏的感知来提升其可靠性的潜力，为在真实自动驾驶环境中开发更值得信赖、更具可解释性的决策系统提供路线图。该基准工具包可公开获取。

英文摘要：

Recent advancements in Vision-Language Models (VLMs) have sparked interest in their use for autonomous driving, particularly in generating interpretable driving decisions through natural language. However, the assumption that VLMs inherently provide visually grounded, reliable, and interpretable explanations for driving remains largely unexamined. To address this gap, we introduce DriveBench, a benchmark dataset designed to evaluate VLM reliability across 17 settings (clean, corrupted, and text-only inputs), encompassing 19,200 frames, 20,498 question-answer pairs, three question types, four mainstream driving tasks, and a total of 12 popular VLMs. Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding, especially under degraded or missing visual inputs. This behavior, concealed by dataset imbalances and insufficient evaluation metrics, poses significant risks in safety-critical scenarios like autonomous driving. We further observe that VLMs struggle with multi-modal reasoning and display heightened sensitivity to input corruptions, leading to inconsistencies in performance. To address these challenges, we propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding. Additionally, we highlight the potential of leveraging VLMs' awareness of corruptions to enhance their reliability, offering a roadmap for developing more trustworthy and interpretable decision-making systems in real-world autonomous driving contexts. The benchmark toolkit is publicly accessible.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。