仲裁失败，而非感知失明：视觉-语言模型如何解决视觉-语言冲突
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
AI总结：
研究探讨视觉-语言模型在视觉与语言冲突中的仲裁机制，发现编码与基础的脱节，通过多模态仲裁交叉分析揭示视觉属性在早期层可线性解码，最终层logit与基础结果相关性达0.847，表明需针对性干预提升视觉基础能力。
摘要：
当视觉-语言模型（VLM）看到一个蓝色的香蕉并回答'黄色'时，问题出在感知还是仲裁？我们通过十种不同规模的VLM探讨了这一问题，揭示了编码与基础的脱节：未能报告所见（从而提供错误答案）的模型仍能强烈地编码视觉证据，与提供正确答案的模型相当。通过多模态仲裁交叉分析（MAC）结合逐层logit镜头探测，我们追踪了每层模型中视觉与先验信号之间的竞争。我们发现，视觉属性可以从早期层线性解码（AUC>0.86）。成功与失败样本的准确性几乎相同。然而，最终层logit之间的差距——而不是编码的强度——更能预测基础结果，相关系数为ρ=0.847。在研究VLMs如何基于图像线索而非先验知识做出回答后，我们希望理解因果关系。我们通过全序列激活补丁处理建立了因果性。标准的最后token干预在LLM可解释性中不起作用。相反，用MAC识别的层替换完整token序列会改变60到84%的输出。部分token分解显示，图像token几乎承担了全部因果影响，而文本token则没有。扩展解决了剩余的架构差异以实现完美保留。从诊断到干预，我们显示在早期层中训练自由的激活引导——无论是线性还是稀疏自编码器引导的——可以将视觉基础能力提高高达+3.8%，在某些设置中性能会下降。总体而言，这些发现得出明确的结论：VLMs已经能很好地看到，但挑战在于如何行动。针对性干预可以帮助弥合这一差距。
英文摘要：
When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of $ρ=$ 0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
