视觉与语言模型中的跨模态关联：重新审视布巴-奇奇效应
Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect
AI总结：
本研究重新评估CLIP的ResNet和ViT变体中的布巴-奇奇效应，发现它们未一致表现出该效应，且与人类数据相比存在显著差距，揭示了VLM跨模态理解的局限性。
摘要：
多模态模型的最新进展引发了这样一个问题：视觉与语言模型（VLMs）是否以反映人类认知的方式整合跨模态信息。该领域中一个被广泛研究的测试案例是布巴-奇奇效应，即人类可靠地将“bouba”这样的伪词与圆形形状相关联，而将“kiki”与锯齿状形状相关联。鉴于先前研究中关于VLMs中该效应的证据不一，我们针对CLIP的两种变体——ResNet和视觉Transformer（ViT）——进行了全面的重新评估，因为它们在许多最先进的VLMs中占据核心地位。我们采用了两种紧密模拟人类实验的互补方法：一种基于提示的评估，使用概率作为模型偏好的度量；以及我们使用Grad-CAM作为一种新颖的方法来解释形状-单词匹配任务中的视觉注意力。我们的研究结果表明，这些模型变体并未一致地表现出布巴-奇奇效应。虽然ResNet表现出对圆形形状的偏好，但两种模型变体的整体表现缺乏预期的关联。此外，与先前人类在同一任务上的数据直接比较显示，模型的响应明显达不到人类认知特有的那种稳健的、模态整合的行为。这些结果有助于当前关于VLMs在多大程度上真正理解跨模态概念的辩论，突显了它们内部表征的局限性及其与人类直觉的一致性。
英文摘要：
Recent advances in multimodal models have raised questions about whether vision-and-language models (VLMs) integrate cross-modal information in ways that reflect human cognition. One well-studied test case in this domain is the bouba-kiki effect, where humans reliably associate pseudowords like `bouba' with round shapes and `kiki' with jagged ones. Given the mixed evidence found in prior studies for this effect in VLMs, we present a comprehensive re-evaluation focused on two variants of CLIP, ResNet and Vision Transformer (ViT), given their centrality in many state-of-the-art VLMs. We apply two complementary methods closely modelled after human experiments: a prompt-based evaluation that uses probabilities as a measure of model preference, and we use Grad-CAM as a novel approach to interpret visual attention in shape-word matching tasks. Our findings show that these model variants do not consistently exhibit the bouba-kiki effect. While ResNet shows a preference for round shapes, overall performance across both model variants lacks the expected associations. Moreover, direct comparison with prior human data on the same task shows that the models' responses fall markedly short of the robust, modality-integrated behaviour characteristic of human cognition. These results contribute to the ongoing debate about the extent to which VLMs truly understand cross-modal concepts, highlighting limitations in their internal representations and alignment with human intuitions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
