幻象探针：视觉模型如何伪造视觉理解
Mirage Probes: How Vision Models Fake Visual Understanding
AI总结：
提出幻象探针框架，通过对比探针揭示视觉语言模型在无图像时也能回答问题的两种幻象行为：文本偏见和虚假图像，并证明后者需要表征级干预。
摘要：
视觉语言模型（VLM）即使在没有提供图像的情况下，也能自信且通常正确地回答基于图像的问题。这种幻象行为会虚增基准分数，而不反映视觉基础。先前的工作将其视为单一故障模式。我们认为这是两种。使用幻象探针（Mirage Probes），一种对比探针框架，将释义的问题变体与同一图像上的匹配幻象和非幻象标签配对，我们展示了在两个开源VLM中，幻象行为可以从残差流、MLP、后注意力和注意力头位置的内部激活中线性解码。我们证明朴素贝叶斯文本基线无法恢复此信号，排除了表面词汇混淆。跨基准可分离性模式，连同一种新颖的先验利用指数（PHI），衡量模型仅从文本中回答的程度，揭示了两种不同的机制：文本偏见，其中模型从语言先验中回答而不涉及视觉表征；以及虚假图像，其中模型在潜在空间中构建虚假视觉内容并像有基础一样回答。这种区别有直接的缓解后果：文本分布清理可以解决第一种机制，但无法触及第二种，因为虚假图像幻象存在于模型的视觉表征中而非文本中。忠实的视觉基础将需要在表征层面进行干预。
英文摘要：
Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
