Robusto-2: 在利马与纽约市对自动驾驶中人类与VLM的基准测试
Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City
AI总结：
本研究通过视觉问答范式，比较人类驾驶员（来自利马和纽约）与视觉语言模型在利马和纽约的驾驶场景中的表现，发现人类与模型在回答事实、评级、反事实和推理四类问题时存在差异，但地理因素影响不显著。
摘要：
随着自动驾驶汽车在国际上不断扩展，并采用多模态系统（如VLM）作为其行动模型的认知骨干，这些系统在新环境（特别是新地理区域的分布外边缘案例场景）中的泛化能力如何？在本文中，我们通过提供全因子分析来研究这一开放问题，分析对象包括利马的人类驾驶员、纽约市的人类驾驶员以及VLM，并向他们展示从利马和纽约市收集的行车记录仪视频，在视觉问答范式下提出各种问题。具体而言，我们选择这两个城市是因为它们是极具挑战性的驾驶地点，目前尚无自动驾驶汽车公司在此运营，并且我们提出的问题涵盖4个类别：事实、评级、反事实和推理。我们发现人类和VLM的回答存在分歧——尽管这受到问题类型的调节，并且人类的回答相似，无论他们来自哪里（利马/纽约）。令人惊讶的是，我们没有发现地理因素对答案（人类或VLM）产生强烈差异，这可能是由于其高度分布外的性质。我们的数据集可在以下网址获取：this https URL
英文摘要：
As Self-Driving Cars continue to expand internationally and use multi-modal systems such as VLMs as a cognitive backbone for their Action models; how well will these systems generalize in new settings, in particular out-of-distribution (OOD) edge-case scenarios in new geographies? In this paper, we study this open question by providing a full factorial analysis with human drivers of Lima, human drivers from New York City, and VLMs and showing them dashcam footage collected from Lima and New York City -- prompting them with a variety of questions under a Visual Question Answering (VQA) paradigm. In particular, we pick these two cities as they are highly challenging driving locations where no Self-Driving Car company currently operates in, and ask questions that span 4 categories: Factual, Ratings, Counterfactual and Reasoning. We find that Humans and VLMs diverge in their responses -- though this is modulated by the type of questions asked, and that Humans answer similarly independent of where they are from (Lima/NYC). To our surprise, we did not find a strong difference in terms of answers (Humans or VLMs) that was modulated by geography, likely due to their high out-of-distribution nature. Our dataset is available at: https://huggingface.co/datasets/Artificio/robusto-2
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
