Illusory VQA：在视错觉场景下基准测试并增强多模态模型
Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions
AI总结：
针对现有VQA数据集忽略视错觉挑战的问题，研究者提出Illusory VQA任务与四个专用数据集，提出基于高斯模糊低通滤波器的预处理方法，可显著提升多模态模型视错觉识别性能，部分场景下超越人类水平。
摘要：
近年来，视觉问答（VQA）领域取得了显著进展，尤其是融合视觉与语言理解的多模态模型出现后更是如此。然而，现有VQA数据集往往忽略了视错觉带来的复杂性，这类错觉对人类感知和模型解读都构成了独特挑战。本研究提出了名为Illusory VQA的新任务，同时构建了四个专用数据集：IllusionMNIST、IllusionFashionMNIST、IllusionAnimals和IllusionChar，用于评估前沿多模态模型识别与解读视错觉的性能。我们评估了多种模型的零样本性能，在自建数据集上对选定模型进行微调，并提出了一种简单有效的错觉检测方案，即使用高斯模糊低通滤波器进行预处理。实验表明，该方法能显著提升模型性能；在IllusionAnimals数据集上，无需微调的BLIP-2结合该方法后表现甚至超过人类。研究结果揭示了人与模型对错觉的感知差异，同时证明微调与特定预处理技术可大幅增强模型鲁棒性。本工作为多模态模型实现更类人的视觉理解提供了支撑，并指明了利用可学习参数适配滤波器的未来研究方向。
英文摘要：
In recent years, Visual Question Answering (VQA) has made significant strides, particularly with the advent of multimodal models that integrate vision and language understanding. However, existing VQA datasets often overlook the complexities introduced by image illusions, which pose unique challenges for both human perception and model interpretation. In this study, we introduce a novel task called Illusory VQA, along with four specialized datasets: IllusionMNIST, IllusionFashionMNIST, IllusionAnimals, and IllusionChar. These datasets are designed to evaluate the performance of state-of-the-art multimodal models in recognizing and interpreting visual illusions. We assess the zero-shot performance of various models, fine-tune selected models on our datasets, and propose a simple yet effective solution for illusion detection using Gaussian and blur low-pass filters. We show that this method increases the performance of models significantly and in the case of BLIP-2 on IllusionAnimals without any fine-tuning, it outperforms humans. Our findings highlight the disparity between human and model perception of illusions and demonstrate that fine-tuning and specific preprocessing techniques can significantly enhance model robustness. This work contributes to the development of more human-like visual understanding in multimodal models and suggests future directions for adapting filters using learnable parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
