视觉语言模型中的特征竞争及其表征起源：来自Stroop范式的证据
Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm
AI总结：
研究人员将Stroop范式改编为基准测试，发现视觉语言模型对Stroop图像中的文本特征存在持续偏向，其表征不对称性导致该偏向，凸显需理解线索显著性对模型下游性能的影响。
摘要：
当你向视觉语言模型（VLM）提出带有图像的问题时，模型会基于图像回答问题，但行动意味着选择：给出一个答案就意味着选择不给出另一个答案。当图像包含多个与当前问题同等相关的特征时，这一问题就会凸显：VLM会如何选择回应？心理学中的经典Stroop任务正是针对这一问题的研究：在一组刻意设计的文本与颜色冲突的图像（例如\textcolor{blue}{RED}）中，要求参与者报告颜色。我们将这一心理学范式改编为基准测试，并对一系列对比式和生成式VLM的行为进行提示测试。在所有模型中，我们观察到Stroop图像中存在对文本特征的持续偏向：当文本与颜色不一致时，模型优先生成以文本为中心而非颜色的回应。随后我们打开模型黑箱，研究这一偏向的来源，以及为何某些特征对模型行为的影响大于其他特征。在嵌入空间中，我们发现文本线索的表征比颜色线索更强烈、更具可分性。这种表征不对称性可预测干预的成功性：引导模型朝向文本相关方向的难度远低于引导其朝向颜色相关方向。我们利用经典心理学范式，在行为和表征层面证明，同一图像中的不同特征可对VLM行为产生不等的影响。研究结果表明，VLM会为同一图像中的不同特征分配不同程度的显著性，凸显了理解特定线索的显著性如何影响其在广泛使用的下游应用中的性能的必要性。
英文摘要：
When you ask a vision-language model (VLM) a question with an image, it answers your question based on the image. But to act means to choose: saying one thing means choosing to respond not say another. This becomes an issue when the image contains multiple features that equally relate to the question at hand: how would VLMs choose to respond? This problem is precisely studied by the classical Stroop task in psychology: among a set of deliberately designed images whose text and color conflicts (e.g., \textcolor{blue}{RED}), participants are instructed to report the color. We adapt this psychological paradigm to a benchmark and prompted a range of contrastive and generative VLMs on their behavior. Across models, we observe a persistent bias toward the feature of text in Stroop images: when text and color disagree, models preferentially generate responses centered to the text rather than color. We then opened the black box and examined the source of this bias and why certain features influences the model's behavior more than others. In the embedding space, we found that textual cues are represented more strongly and separably than color cues. This representational asymmetry predicts intervention success: steering models toward text-related directions is substantially easier than steering them toward color-related directions. Using a classical psychological paradigm, we demonstrate, at both behavioral and representational levels, that distinct features within the same image can exert unequal influence on VLM behavior. Our findings suggest that VLMs assign different degrees of saliency to different features within the same image, highlighting the need to understand how cue-specific saliency shapes their performance in widely used downstream applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
