本能性偏差：伪相关图像导致多模态大语言模型产生错觉
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
AI总结：
本文提出首个评估多模态大语言模型视觉错觉的基准CorrelationQA，包含13类7308个文本-图像对，并通过9个主流模型的实验揭示了MLLMs普遍存在由伪相关图像引发的本能性偏差。
摘要：
大语言模型（LLMs）近年来取得了显著进展，多模态大语言模型（MLLMs）的出现赋予了LLMs视觉能力，使其在各类多模态任务中表现出色。然而，像GPT-4V这样强大的MLLMs在遇到某些图像和文本输入时仍会出现严重失败。本文识别出一类典型的会使MLLMs产生困惑的输入，这类输入包含与答案高度相关但不一致的图像，导致MLLMs遭受视觉错觉。为了量化这种效应，我们提出了CorrelationQA，这是首个在给定伪相关图像条件下评估视觉错觉水平的基准。该基准包含13个类别的7,308个文本-图像对。基于所提出的CorrelationQA，我们对9个主流MLLMs进行了全面分析，结果表明它们普遍在不同程度上受到这种本能性偏差的影响。我们希望我们所构建的基准和评估结果有助于更好地评估MLLMs在存在误导性图像时的鲁棒性。代码和数据集可在https://github.com/MasaiahHan/CorrelationQA获取。
英文摘要：
Large language models (LLMs) have recently experienced remarkable progress, where the advent of multi-modal large language models (MLLMs) has endowed LLMs with visual capabilities, leading to impressive performances in various multi-modal tasks. However, those powerful MLLMs such as GPT-4V still fail spectacularly when presented with certain image and text inputs. In this paper, we identify a typical class of inputs that baffles MLLMs, which consist of images that are highly relevant but inconsistent with answers, causing MLLMs to suffer from visual illusion. To quantify the effect, we propose CorrelationQA, the first benchmark that assesses the visual illusion level given spurious images. This benchmark contains 7,308 text-image pairs across 13 categories. Based on the proposed CorrelationQA, we conduct a thorough analysis on 9 mainstream MLLMs, illustrating that they universally suffer from this instinctive bias to varying degrees. We hope that our curated benchmark and evaluation results aid in better assessments of the MLLMs' robustness in the presence of misleading images. The code and datasets are available at https://github.com/MasaiahHan/CorrelationQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
