arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.07587cs.CVcs.AIcs.RO
收藏

Robusto-1 数据集:在秘鲁真实分布外自动驾驶 VQA 中比较人类与视觉语言模型

Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Peru

Artificio

Dunant Cusipuma, David Ortega, Victor Flores-Benites, Arturo Deza

AI总结:

本文构建秘鲁真实分布外自动驾驶 VQA 数据集 Robusto-1,并利用表征相似性分析比较人类与视觉语言模型的认知对齐,发现对齐程度因问题类型不同而显著变化。

摘要:

随着多模态基础模型开始在自动驾驶汽车中进行实验性部署,我们向自己提出一个合理的问题:在某些驾驶情境中——尤其是那些分布外的情境——这些系统的反应与人类有多相似?为了研究这一问题,我们构建了 Robusto-1 数据集,该数据集使用来自秘鲁的行车记录仪视频数据。秘鲁拥有世界上最差(激进)的驾驶员之一、较高的交通指数,以及高比例的训练中可能从未见过的奇异与非奇异街道物体。特别地,为了在认知层面初步测试基础视觉语言模型(VLMs)在驾驶中与人类的比较情况,我们从边界框、分割图、占用图或轨迹估计转向多模态视觉问答(VQA),通过系统神经科学中一种流行的方法——表征相似性分析(RSA)——来比较人类和机器。根据我们所提问题的类型以及这些系统给出的答案,我们将展示在哪些情况下 VLMs 与人类收敛或发散,从而使我们能够探究它们的认知对齐程度。我们发现,对齐程度根据向每种系统(人类与 VLMs)所提问题的类型而显著不同,这凸显了它们在认知对齐上的差距。

英文摘要:

As multimodal foundational models start being deployed experimentally in Self-Driving cars, a reasonable question we ask ourselves is how similar to humans do these systems respond in certain driving situations -- especially those that are out-of-distribution? To study this, we create the Robusto-1 dataset that uses dashcam video data from Peru, a country with one of the worst (aggressive) drivers in the world, a high traffic index, and a high ratio of bizarre to non-bizarre street objects likely never seen in training. In particular, to preliminarly test at a cognitive level how well Foundational Visual Language Models (VLMs) compare to Humans in Driving, we move away from bounding boxes, segmentation maps, occupancy maps or trajectory estimation to multi-modal Visual Question Answering (VQA) comparing both humans and machines through a popular method in systems neuroscience known as Representational Similarity Analysis (RSA). Depending on the type of questions we ask and the answers these systems give, we will show in what cases do VLMs and Humans converge or diverge allowing us to probe on their cognitive alignment. We find that the degree of alignment varies significantly depending on the type of questions asked to each type of system (Humans vs VLMs), highlighting a gap in their alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑