视觉语言模型能成为好的猜测者吗？探索视觉语言模型在时间与地点推理中的能力
Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning
AI总结：
本文提出两阶段识别与推理探测任务及WikiTiLo数据集，探究视觉语言模型在时间与地点推理上的能力，发现其虽能保留特征但推理不完美。
摘要：
视觉语言模型（VLMs）被期望能够像人类一样运用常识知识进行推理。一个例子是，人类可以根据自己的知识推断出图像拍摄的地点和时间。这让我们不禁思考：基于视觉线索，经过大规模图像-文本资源预训练的视觉语言模型，是否能够达到甚至超越人类在时间和地点推理方面的能力？为了回答这个问题，我们提出了一个两阶段的“识别与推理”探测任务，应用于判别式和生成式视觉语言模型，以揭示视觉语言模型是否能够识别与时间和地点相关的特征，并进一步进行推理。为了促进这项研究，我们引入了WikiTiLo，一个精心整理的图像数据集，其中包含具有丰富社会文化线索的图像。在广泛的实验研究中，我们发现，尽管视觉语言模型能够有效地在视觉编码器中保留相关特征，但它们仍然无法进行完美的推理。我们将发布我们的数据集和代码，以促进未来的研究。
英文摘要：
Vision-Language Models (VLMs) are expected to be capable of reasoning with commonsense knowledge as human beings. One example is that humans can reason where and when an image is taken based on their knowledge. This makes us wonder if, based on visual cues, Vision-Language Models that are pre-trained with large-scale image-text resources can achieve and even outperform human's capability in reasoning times and location. To address this question, we propose a two-stage \recognition\space and \reasoning\space probing task, applied to discriminative and generative VLMs to uncover whether VLMs can recognize times and location-relevant features and further reason about it. To facilitate the investigation, we introduce WikiTiLo, a well-curated image dataset compromising images with rich socio-cultural cues. In the extensive experimental studies, we find that although VLMs can effectively retain relevant features in visual encoders, they still fail to make perfect reasoning. We will release our dataset and codes to facilitate future studies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
