Chimera：诊断视觉-语言理解中的捷径学习
Chimera: Diagnosing Shortcut Learning in Visual-Language Understanding
AI总结：
本文提出 Chimera 测试套件，通过 7,500 张标注图表诊断视觉-语言模型中的三类捷径行为，发现模型表现主要源于 Clever-Hans 捷径而非真正理解，揭示了当前 VLMs 的关键局限。
摘要：
图表以视觉形式而非线性文字流传递符号信息，这使得人工智能模型处理起来尤为困难。尽管近期评估表明视觉-语言模型（VLMs）在图表相关基准上表现良好，但它们对知识、推理或模态捷径的依赖引发了对其是否真正理解并推理图表的担忧。为填补这一空白，我们引入了 Chimera，一个包含 7,500 张来自维基百科的高质量图表的综合测试套件；每张图表都标注了以语义三元组表示的符号内容，并配有旨在评估图表理解四个基本方面的多层级问题：实体识别、关系理解、知识 grounding 和视觉推理。我们使用 Chimera 来测量视觉问答中三类捷径的存在：（1）视觉记忆捷径，即 VLMs 依赖记忆化的视觉模式；（2）知识回忆捷径，即模型利用记忆化的事实知识而非解读图表；（3）Clever-Hans 捷径，即模型在没有真正理解的情况下利用表层语言模式或先验。我们在 Chimera 上评估了来自 7 个模型家族的 15 个开源 VLMs，发现它们看似强劲的表现主要源于捷径行为：视觉记忆捷径影响轻微，知识回忆捷径作用中等，而 Clever-Hans 捷径贡献显著。这些发现揭示了当前 VLMs 的关键局限，并强调需要更稳健的评估协议，以基准测试对复杂视觉输入（如图表）的真正理解，而非问答捷径。
英文摘要：
Diagrams convey symbolic information in a visual format rather than a linear stream of words, making them especially challenging for AI models to process. While recent evaluations suggest that vision-language models (VLMs) perform well on diagram-related benchmarks, their reliance on knowledge, reasoning, or modality shortcuts raises concerns about whether they genuinely understand and reason over diagrams. To address this gap, we introduce Chimera, a comprehensive test suite comprising 7,500 high-quality diagrams sourced from Wikipedia; each diagram is annotated with its symbolic content represented by semantic triples along with multi-level questions designed to assess four fundamental aspects of diagram comprehension: entity recognition, relation understanding, knowledge grounding, and visual reasoning. We use Chimera to measure the presence of three types of shortcuts in visual question answering: (1) the visual-memorization shortcut, where VLMs rely on memorized visual patterns; (2) the knowledge-recall shortcut, where models leverage memorized factual knowledge instead of interpreting the diagram; and (3) the Clever-Hans shortcut, where models exploit superficial language patterns or priors without true comprehension. We evaluate 15 open-source VLMs from 7 model families on Chimera and find that their seemingly strong performance largely stems from shortcut behaviors: visual-memorization shortcuts have slight impact, knowledge-recall shortcuts play a moderate role, and Clever-Hans shortcuts contribute significantly. These findings expose critical limitations in current VLMs and underscore the need for more robust evaluation protocols that benchmark genuine comprehension of complex visual inputs (e.g., diagrams) rather than question-answering shortcuts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
