arXiv 2501.008482025-06-19cs.CV

IllusionBench+：面向视觉语言模型的视觉错觉理解的大规模综合基准

IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models

Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, China（图像通信与网络工程研究院，上海交通大学）

Yiming Zhang, Zicheng Zhang, Xinyi Wei, Xiaohong Liu, Guangtao Zhai, Xiongkuo Min

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有基准忽略真实世界视觉错觉的问题，提出包含经典与真实错觉的IllusionBench数据集，含1,051图像及多任务评估，揭示SOTA模型如GPT-4o仍落后于人类并存在幻觉。

摘要：

当前的视觉语言模型（VLMs）在图像理解方面表现出色，但在视觉错觉方面，尤其是在真实世界场景中，仍存在困难。现有基准侧重于经典认知错觉，而这些已被最先进的（SOTA）VLMs学习，揭示了幻觉和有限感知能力等问题。为弥补这一空白，我们引入了IllusionBench，一个全面的视觉错觉数据集，不仅包含经典认知错觉，还涵盖真实世界场景错觉。该数据集包含1,051张图像、5,548个问答对以及1,051条金色文本描述，这些描述涉及错觉的存在、成因和内容。我们使用真/假、多项选择和开放式任务在该数据集上评估了十个SOTA VLMs。除真实世界错觉外，我们还设计了陷阱错觉，它们类似于经典模式但在现实中不同，从而凸显SOTA模型中的幻觉问题。表现最佳的模型GPT-4o在真/假任务上达到80.59%的准确率，在多项选择题上达到76.75%，但仍落后于人类表现。在语义描述任务中，GPT-4o对经典错觉的幻觉导致其在陷阱错觉上得分较低，甚至落后于一些开源模型。据我们所知，IllusionBench是迄今为止针对VLMs中视觉错觉的最大且最全面的基准。

英文摘要：

Current Visual Language Models (VLMs) show impressive image understanding but struggle with visual illusions, especially in real-world scenarios. Existing benchmarks focus on classical cognitive illusions, which have been learned by state-of-the-art (SOTA) VLMs, revealing issues such as hallucinations and limited perceptual abilities. To address this gap, we introduce IllusionBench, a comprehensive visual illusion dataset that encompasses not only classic cognitive illusions but also real-world scene illusions. This dataset features 1,051 images, 5,548 question-answer pairs, and 1,051 golden text descriptions that address the presence, causes, and content of the illusions. We evaluate ten SOTA VLMs on this dataset using true-or-false, multiple-choice, and open-ended tasks. In addition to real-world illusions, we design trap illusions that resemble classical patterns but differ in reality, highlighting hallucination issues in SOTA models. The top-performing model, GPT-4o, achieves 80.59% accuracy on true-or-false tasks and 76.75% on multiple-choice questions, but still lags behind human performance. In the semantic description task, GPT-4o's hallucinations on classical illusions result in low scores for trap illusions, even falling behind some open-source models. IllusionBench is, to the best of our knowledge, the largest and most comprehensive benchmark for visual illusions in VLMs to date.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。