视觉语言模型能否从视觉参考中回忆事实关联？
Can VLMs Recall Factual Associations From Visual References?
AI总结：
研究发现视觉语言模型在视觉参考下回忆事实关联能力减半，通过内部状态探针可高精度检测失败，提升选择性预测覆盖率并降低错误率。
摘要：
通过一项受控研究，我们识别出视觉语言模型（VLMs）在多模态接地中存在系统性缺陷。当提供实体的文本参考时，VLMs能够回忆事实关联；但当参考变为视觉形式时，其能力显著下降。迫使VLMs依赖实体的图像表示会使其回忆事实知识的能力减半，这表明VLMs难以将其实体的内部知识与图像表示联系起来。我们表明，这种连接失败与模型内部状态中不同模式的表达相关，且基于这些内部状态的探针在标记VLM响应不可靠的情况时，准确率超过92%。这些探针无需重新训练即可应用，用于识别VLM在需要理解多模态输入的问题上何时会无法正确回答。当用于促进视觉问答任务中的选择性预测时，探针将覆盖率提高了7.87%（绝对值），同时将错误风险降低了0.9%（绝对值）。解决这一系统性、可检测的缺陷是语言接地的重要途径，我们为未来方向提供了有依据的建议。
英文摘要：
Through a controlled study, we identify a systematic deficiency in the multimodal grounding of Vision Language Models (VLMs). While VLMs can recall factual associations when provided a textual reference to an entity; their ability to do so is significantly diminished when the reference is visual instead. Forcing VLMs to rely on image representations of an entity halves their ability to recall factual knowledge, suggesting that VLMs struggle to link their internal knowledge of an entity with its image representation. We show that such linking failures are correlated with the expression of distinct patterns in model internal states, and that probes on these internal states achieve over 92% accuracy at flagging cases where the VLM response is unreliable. These probes can be applied, without retraining, to identify when a VLM will fail to correctly answer a question that requires an understanding of multimodal input. When used to facilitate selective prediction on a visual question answering task, the probes increase coverage by 7.87% (absolute) while also reducing the risk of error by 0.9% (absolute). Addressing the systematic, detectable deficiency is an important avenue in language grounding, and we provide informed recommendations for future directions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
