看见即相信？多模态大语言模型在视觉错觉和异常上的基准测试
Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies
AI总结：
本文提出VIA-Bench基准测试，评估多模态大语言模型在视觉错觉和异常上的性能，揭示其在复杂视觉任务中的鲁棒性问题。
摘要：
多模态大语言模型（MLLMs）在通用视觉语言基准测试中表现出色，达到甚至超越了人类水平。然而，这些评估通常依赖于标准分布数据，当面对违反常识先验的场景时，MLLMs的鲁棒性仍缺乏充分检验。为填补这一空白，我们引入了VIA-Bench，一个旨在探测模型在视觉错觉和异常上的性能的挑战性基准。它包括六个核心类别：颜色错觉、运动错觉、格式错觉、几何和空间错觉、一般视觉错觉以及视觉异常。通过仔细的人工循环审查，我们构建了超过1000对高质量问题-答案对，这些需要细致的视觉推理。对超过20种最先进的MLLMs进行了广泛评估，包括专有、开源和推理增强模型，揭示了显著的脆弱性。值得注意的是，我们发现链式推理（CoT）推理提供微不足道的鲁棒性，常常产生“脆弱幻象”，即模型的逻辑在错觉刺激下崩溃。我们的发现揭示了机器与人类感知之间的根本分歧，表明解决此类感知瓶颈对于人工智能通用智能的发展至关重要。基准数据和代码将被发布。
英文摘要：
Multimodal Large Language Models (MLLMs) have shown remarkable proficiency on general-purpose vision-language benchmarks, reaching or even exceeding human-level performance. However, these evaluations typically rely on standard in-distribution data, leaving the robustness of MLLMs largely unexamined when faced with scenarios that defy common-sense priors. To address this gap, we introduce VIA-Bench, a challenging benchmark designed to probe model performance on visual illusions and anomalies. It includes six core categories: color illusions, motion illusions, gestalt illusions, geometric and spatial illusions, general visual illusions, and visual anomalies. Through careful human-in-the-loop review, we construct over 1K high-quality question-answer pairs that require nuanced visual reasoning. Extensive evaluation of over 20 state-of-the-art MLLMs, including proprietary, open-source, and reasoning-enhanced models, uncovers significant vulnerabilities. Notably, we find that Chain-of-Thought (CoT) reasoning offers negligible robustness, often yielding ``brittle mirages'' where the model's logic collapses under illusory stimuli. Our findings reveal a fundamental divergence between machine and human perception, suggesting that resolving such perceptual bottlenecks is critical for the advancement of artificial general intelligence. The benchmark data and code will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
