像素与先验：通过视觉反事实控制视觉-语言模型中的知识先验
Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts
AI总结：
本文构建 Visual CounterFact 冲突数据集揭示 MLLM 中知识先验与视觉证据的层间竞争，并提出 PvP 激活干预向量，以可控地将预测导向先验或视觉反事实。
摘要：
多模态大语言模型（MLLM）在视觉问答等任务上表现良好，但其推理究竟更多依赖记忆的世界知识，还是输入图像中呈现的视觉信息，仍不清楚。为研究这一点，我们引入 Visual CounterFact，这是一个包含视觉真实反事实样本的新数据集，使世界知识先验（例如红色草莓）与视觉输入（例如蓝色草莓）直接冲突。利用 Visual CounterFact，我们表明模型预测最初反映记忆的先验，但在中间到后期层会转向视觉证据。这一动态揭示了两种模态之间的竞争，并且在评估过程中视觉输入最终覆盖先验。为控制该行为，我们提出 Pixels Versus Priors（PvP）引导向量，这是一种通过激活层面干预，将模型输出导向世界知识或视觉输入的机制。平均而言，PvP 成功将 99.3% 的颜色预测和 80.8% 的尺寸预测从先验转向反事实。总之，这些发现为解释和控制多模态模型中的事实行为提供了新工具。
英文摘要：
Multimodal Large Language Models (MLLMs) perform well on tasks such as visual question answering, but it remains unclear whether their reasoning relies more on memorized world knowledge or on the visual information present in the input image. To investigate this, we introduce Visual CounterFact, a new dataset of visually-realistic counterfactuals that put world knowledge priors (e.g, red strawberry) into direct conflict with visual input (e.g, blue strawberry). Using Visual CounterFact, we show that model predictions initially reflect memorized priors, but shift toward visual evidence in mid-to-late layers. This dynamic reveals a competition between the two modalities, with visual input ultimately overriding priors during evaluation. To control this behavior, we propose Pixels Versus Priors (PvP) steering vectors, a mechanism for controlling model outputs toward either world knowledge or visual input through activation-level interventions. On average, PvP successfully shifts 99.3% of color and 80.8% of size predictions from priors to counterfactuals. Together, these findings offer new tools for interpreting and controlling factual behavior in multimodal models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
