arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.12969cs.CLcs.CVcs.LG
收藏

透过他们的眼睛：评估视觉语言模型中的视觉视角采择

Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models

University of Warsaw（华沙大学）；IDEAS NCBR；IMPAN（波兰科学院数学研究所）

Gracjan Góral, Alicja Ziarko, Michal Nauman, Maciej Wołczyk

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究视觉语言模型（VLMs）的视觉视角采择（VPT）能力，引入Isle-Bricks和Isle-Dots数据集评估12个常用VLM，发现模型在需视角采择时性能显著下降，且目标检测性能与VPT性能相关性弱。

摘要：

视觉视角采择（VPT）是理解他人视角的能力，使个体能够预判他人的行动。例如，驾驶员可以通过评估行人看到的内容来避免事故。人类通常在幼儿期就发展出这种技能，但目前尚不清楚近期涌现的视觉语言模型（VLMs）是否具备这种能力。此外，随着这些模型越来越多地部署在现实世界中，理解它们如何执行像VPT这样精细的任务变得至关重要。在本文中，我们引入了两个手动策划的数据集Isle-Bricks和Isle-Dots来测试VPT技能，并使用它们评估了12个常用的VLM。在所有模型中，我们观察到当需要视角采择时，性能显著下降。此外，我们发现目标检测任务的性能与VPT任务的性能相关性很差，这表明现有的基准可能不足以理解这一问题。代码和数据集将发布于https://sites.google.com/view/perspective-taking

英文摘要：

Visual perspective-taking (VPT), the ability to understand the viewpoint of another person, enables individuals to anticipate the actions of other people. For instance, a driver can avoid accidents by assessing what pedestrians see. Humans typically develop this skill in early childhood, but it remains unclear whether the recently emerging Vision Language Models (VLMs) possess such capability. Furthermore, as these models are increasingly deployed in the real world, understanding how they perform nuanced tasks like VPT becomes essential. In this paper, we introduce two manually curated datasets, Isle-Bricks and Isle-Dots for testing VPT skills, and we use it to evaluate 12 commonly used VLMs. Across all models, we observe a significant performance drop when perspective-taking is required. Additionally, we find performance in object detection tasks is poorly correlated with performance on VPT tasks, suggesting that the existing benchmarks might not be sufficient to understand this problem. The code and the dataset will be available at https://sites.google.com/view/perspective-taking

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑