看见但不相信：探究 VLM 中视觉注意力与答案正确性之间的脱节
Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
AI总结：
本文通过逐层注意力分析揭示 VLM 常已感知正确视觉证据却仍答错，并提出无需训练的推理时选择性注意力掩蔽干预，在多类模型上持续提升准确率。
摘要：
视觉语言模型（Vision-Language Models，VLMs）在视觉问答等多模态任务上取得了强劲结果，但即使正确的视觉证据存在，它们仍可能失败。在本工作中，我们系统地研究这些失败究竟源于未能感知证据，还是源于未能有效利用证据。通过检查逐层注意力动态，我们发现浅层主要关注文本，而深层会稀疏但可靠地注意局部证据区域。令人惊讶的是，VLM 在输出错误答案时往往已经感知到视觉证据，我们将这种现象称为“看见但不相信”，并发现它广泛存在于主要 VLM 系列中。基于此，我们提出一种推理时干预方法，通过基于选择性注意力的掩蔽来突出深层证据区域。该方法无需训练，并在 LLaVA、Qwen、Gemma 和 InternVL 等多个模型系列中持续提升准确率。这些结果表明，VLM 在内部编码了可靠证据，却未充分利用这些证据；使此类信号显式化可以弥合感知与推理之间的差距，从而推进对 VLM 的诊断理解并提高其可靠性。
英文摘要：
Vision-Language Models (VLMs) achieve strong results on multimodal tasks such as visual question answering, yet they can still fail even when the correct visual evidence is present. In this work, we systematically investigate whether these failures arise from not perceiving the evidence or from not leveraging it effectively. By examining layer-wise attention dynamics, we find that shallow layers focus primarily on text, while deeper layers sparsely but reliably attend to localized evidence regions. Surprisingly, VLMs often perceive the visual evidence when outputting incorrect answers, a phenomenon we term ``seeing but not believing'' that widely exists in major VLM families. Building on this, we introduce an inference-time intervention that highlights deep-layer evidence regions through selective attention-based masking. It requires no training and consistently improves accuracy across multiple families, including LLaVA, Qwen, Gemma, and InternVL. These results show that VLMs encode reliable evidence internally but under-utilize it, making such signals explicit can bridge the gap between perception and reasoning, advancing the diagnostic understanding and reliability of VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
