arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.16634cs.ROcs.CV
收藏

Language2Drive：视觉-语言-动作模型在自动驾驶中是否真的遵循指令？

Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?

Zihan You, Zehua Fan, Wenqi Lyu, Aoqi Wang, Ni Wang, Yifei Yang, Linge Zhao, Huan Li, Sining Ang, Yuguang Yang, Jueyu Chen, Xiangyu Li, Qianshan Wei, Jieyuan Pe… 展开作者

Zihan You, Zehua Fan, Wenqi Lyu, Aoqi Wang, Ni Wang, Yifei Yang, Linge Zhao, Huan Li, Sining Ang, Yuguang Yang, Jueyu Chen, Xiangyu Li, Qianshan Wei, Jieyuan Pei, Yuxin Wang, Zhiwen Chen, Jiaqi Li, Wenxuan Song, Yanyuan Qiao, Benjin Zhu, Haoang Li, Yan Wang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究针对自动驾驶VLA模型，构建Language2Drive基准，发现其93.2%成功运行在反事实指令下无对应成功结果，暴露现有评估及模型的指令遵循能力局限。

摘要：

近期，视觉-语言-动作（VLA）模型越来越多地使用自然语言指令来约束自动驾驶行为。然而，现有的闭环基准主要评估路线完成度、安全性和规则合规性，留下了一个根本问题未解决：语言是否真的决定了车辆执行的可行行为？一条轨迹可能安全且成功，却仍违反甚至忽略了乘客的指令。为解决这一问题，我们引入了Language2Drive（L2D），一个旨在直接测试该能力的基准。其核心评估单元是配对反事实单元（PCU），它固定初始驾驶场景，仅改变指令。该设计使我们能够在四个指令遵循范围内区分孤立指令成功与反事实配对成功。为测试该区分是否在不同评估场景中持续存在，我们在互补的闭环CARLA仿真、重建的真实场景以及开环驾驶基准中实例化了Language2Drive，包括我们发布的带有专用地图的CARLA-Language2Drive仿真器。在这些设置中评估代表性驾驶VLA后，我们发现了一个显著的不匹配：强大的驾驶能力与薄弱的语言控制之间存在差距：93.2%的成功运行在替代可行指令下没有对应的成功结果。这种失败主要由单侧完成导致，即模型反复执行一种偏好行为，但在同一场景需要反事实替代时失败。这些结果揭示了当前评估的两个根本局限：传统驾驶基准广泛使用的孤立成功和驾驶分数会大幅高估语言遵循能力，而当前的驾驶VLA本身在匹配的反事实测试下缺乏可靠的指令遵循能力。

英文摘要：

Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing closed-loop benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving benchmarks, including our released CARLA-Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving benchmarks, can substantially overstate language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑