VLMs需要词语：视觉语言模型倾向于通过语义锚点而非视觉细节进行推理
VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
AI总结：
研究发现视觉语言模型在需要细粒度视觉感知的任务中表现不佳，因其依赖语义锚点而非视觉细节。通过实验验证，模型在可命名实体上表现更佳，而Logit Lens分析显示其能恢复语义标签。任务特定微调可提升性能，无需语言先验。
摘要：
视觉语言模型（VLMs）在多种多模态任务中表现出色，但在需要细粒度视觉感知的任务中却常常失败，即使所需信息仍存在于内部表示中。先前研究将这一'显而易见'的差距归因于语言模型，但原因仍未明了。本文证明，这一差距源于语言模型缺乏对细粒度视觉细节的语义标签：当视觉实体可映射到已知概念时，VLMs会绕过视觉比较而通过语言进行推理；当无法映射时，VLMs会依赖脆弱且幻觉化的描述。我们通过语义对应、合成形状匹配和人脸匹配验证了这一现象，并发现当相关实体可命名时，VLMs表现更好。机制上，Logit Lens分析确认VLMs会显式恢复可命名实体的语义标签，并在不可命名实体上生成更多独特标记。此外，我们证明这一限制可通过教授完全任意的名称来解决。更重要的是，任务特定微调可带来更强的泛化能力，无需依赖语言先验，即通过真实视觉感知。我们的发现表明，当前VLM在视觉任务中的失败反映的是学习到的捷径，而非多模态推理的根本限制。
英文摘要：
Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they often fail on tasks that require fine-grained visual perception, even when the required information is still present in their internal representations. Prior work has attributed this ``hidden-in-plain-sight'' gap to the language model, but the cause remains unexplained. In this work, we demonstrate that this gap arises from the language model's lack of semantic labels for fine-grained visual details: when visual entities can be mapped to known concepts, VLMs bypass visual comparison and reason through language; when they cannot, VLMs resort to brittle and hallucinated descriptions. We verify this across semantic correspondence, synthetic shape matching, and face matching, and find that VLMs perform much better when the relevant entities are nameable than when they are unnamable. Mechanistically, Logit Lens analysis confirms that VLMs explicitly recover semantic labels for nameable entities and surface more unique tokens compared to unnameable entities. Furthermore, we show that this limitation can be addressed: teaching completely arbitrary names for unknown entities improves performance. More importantly, task-specific finetuning yields even stronger generalization without relying on language priors, i.e., through real visual perception. Our findings suggest that current VLM failures on visual tasks reflect a learned shortcut rather than a fundamental limitation of multimodal reasoning. Code and datasets are available at https://github.com/Patchwork53/VLMs-Need-Words-COLM2026.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
