IllusionVQA：一个面向视觉语言模型的具有挑战性的视错觉数据集
IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models
AI总结：
本文构建IllusionVQA视错觉多选VQA数据集，通过理解与软定位任务揭示GPT4V等VLM与人类的显著差距及ICL、CoT中的定位弱点。
摘要：
视觉语言模型（VLM）的出现使研究人员能够利用自然语言研究神经网络的视觉理解能力。除了物体分类和检测之外，VLM还具备视觉理解和常识推理能力。这自然引出了一个问题：当图像本身在本质上不合理时，VLM会如何反应？为此，我们提出了IllusionVQA：一个多样化的数据集，包含具有挑战性的视错觉和难以解释的场景，用于在两种不同的多选VQA任务——理解和软定位——中测试VLM的能力。表现最好的VLM GPT4V在理解任务上达到62.99%的准确率（4-shot），在定位任务上达到49.7%的准确率（4-shot和思维链）。人类评估显示，人类在理解和定位方面分别达到91.03%和100%的准确率。我们发现，上下文学习（ICL）和思维链推理会显著降低Gemini-Pro在定位任务中的性能。附带地，我们发现VLM的ICL能力存在一个潜在弱点：即使正确答案作为少样本示例位于上下文窗口中，它们也无法定位视错觉。
英文摘要：
The advent of Vision Language Models (VLM) has allowed researchers to investigate the visual understanding of a neural network using natural language. Beyond object classification and detection, VLMs are capable of visual comprehension and common-sense reasoning. This naturally led to the question: How do VLMs respond when the image itself is inherently unreasonable? To this end, we present IllusionVQA: a diverse dataset of challenging optical illusions and hard-to-interpret scenes to test the capability of VLMs in two distinct multiple-choice VQA tasks - comprehension and soft localization. GPT4V, the best performing VLM, achieves 62.99% accuracy (4-shot) on the comprehension task and 49.7% on the localization task (4-shot and Chain-of-Thought). Human evaluation reveals that humans achieve 91.03% and 100% accuracy in comprehension and localization. We discover that In-Context Learning (ICL) and Chain-of-Thought reasoning substantially degrade the performance of Gemini-Pro in the localization task. Tangentially, we discover a potential weakness in the ICL capabilities of VLMs: they fail to locate optical illusions even when the correct answer is in the context window as a few-shot example.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
