真实图像，更差判断：评估视觉-语言模型在具体性和意象性上的表现
Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
AI总结：
本文通过具体性和意象性评分任务，发现真实图像上下文不仅未提升视觉-语言模型与人类判断的一致性，反而在视觉证据相关性低时加剧偏差，并表明推理时聚焦文本指令可缓解退化。
摘要：
视觉输入通常被认为能改善多模态模型的语言理解。我们通过询问视觉-语言模型（VLM）在词汇判断中能否区分有用的视觉证据与偶然的图像上下文来检验这一假设。我们使用人类的具体性和意象性评分，因为它们涵盖从抽象低意象词到具体高意象词等不同视觉相关性的词汇。我们发现，真实图像上下文并未带来一致的增益，反而常常损害与人类评分的一致性，尤其是在视觉证据最不相关时。通过探针分析和典型相关分析，辅以归因案例研究，我们发现真实图像上下文与表征偏移和对虚假视觉线索的更高敏感性相关，同时目标词汇属性的可恢复性减弱。我们进一步表明，在推理时指示模型仅关注文本内容可以减少这种退化，尤其是在这些脆弱子集上效果最明显。我们的发现表明，当前指令微调的VLM需要更好地校准视觉上下文何时应影响词汇判断。
英文摘要：
Visual inputs are often assumed to improve language understanding in multimodal models. We examine this assumption by asking whether vision-language models (VLMs) can distinguish useful visual evidence from incidental image context in lexical judgments. We use human concreteness and imagery ratings because they span words with varying expected visual relevance, from abstract and low-imagery words to concrete and high-imagery words. We find that real-image contexts do not yield consistent gains and often hurt alignment with human ratings, most sharply when visual evidence is least relevant. Through probing and canonical correlation analysis, complemented by an attribution case study, we find that real-image contexts are associated with representational shifts and greater sensitivity to spurious visual cues, coinciding with weaker recoverability of the targeted lexical properties. We further show that instructing models to focus solely on textual content at inference time can reduce this degradation, with the clearest gains on these vulnerable subsets. Our findings suggest that current instruction-tuned VLMs need better calibration of when visual context should inform lexical judgments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
