arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.01259cs.RO
收藏

Bench2Drive-VL: 用于基于视觉语言模型的闭环自动驾驶的基准测试

Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

Shanghai Key Laboratory of Multimodal Embodied AI(上海市多模态具身智能重点实验室)

Xiaosong Jia, Yuqian Shao, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, Junchi Yan

AI总结:

本文提出Bench2Drive-VL,通过闭环评估方法提升视觉语言模型在自动驾驶中的性能,包含DriveCommenter生成多样化问题对、统一协议接口、灵活推理框架及完整开发生态,开源代码和标注数据。

摘要:

随着视觉语言模型(VLM)的兴起,其在自动驾驶(VLM4AD)中的应用受到广泛关注。在自动驾驶领域,闭环评估因其能评估模型在累积误差和非分布输入下的性能,而被认为比开环评估更可靠。然而,现有VLM4AD基准测试仅通过静态问答数据集评估模型的场景理解能力,无法评估在罕见的非分布状态下VLM的表现。为此,我们提出了Bench2Drive-VL,作为Bench2Drive的扩展,引入了DriveCommenter,一种闭环生成器,可自动生成多样化、行为基础的问题-答案对,涵盖CARLA中的所有驾驶情况,包括之前难以评估的严重偏离路线和非道路偏离。此外,还包含统一的协议和接口,允许现代VLM直接接入Bench2Drive闭环环境,与传统智能体进行比较;灵活的推理和控制框架,支持多格式视觉输入和可配置的基于图的推理链执行;以及完整的开发生态系统。这些组件共同构成了一个全面的闭环基准测试,用于VLM4AD。所有代码和标注数据集均已开源。

英文摘要:

With the rise of vision-language models (VLM), their application for autonomous driving (VLM4AD) has gained significant attention. Meanwhile, in autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation, as it can evaluate the performance of the model under cumulative errors and out-of-distribution inputs. However, existing VLM4AD benchmarks evaluate the model`s scene understanding ability under open-loop, i.e., via static question-answer (QA) dataset. This kind of evaluation fails to assess the VLMs performance under out-of-distribution states rarely appeared in the human collected datasets.To this end, we present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving, which introduces: (1) DriveCommenter, a closed-loop generator that automatically generates diverse, behavior-grounded question-answer pairs for all driving situations in CARLA,including severe off-route and off-road deviations previously unassessable in simulation. (2) A unified protocol and interface that allows modern VLMs to be directly plugged into the Bench2Drive closed-loop environment to compare with traditional agents. (3) A flexible reasoning and control framework, supporting multi-format visual inputs and configurable graph-based chain-of-thought execution. (4) A complete development ecosystem. Together, these components form a comprehensive closed-loop benchmark for VLM4AD. All codes and annotated datasets are open sourced.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑