从单个物体中推断上下文在视觉-语言模型中的研究
Contextual inference from single objects in Vision-Language models
AI总结：
本文研究了视觉-语言模型中单个物体推断场景上下文的能力，发现物体属性影响推断性能，且不同层次的推断存在解耦现象。
摘要：
如何确定单个物体所携带的场景上下文是人类场景感知中的经典问题，但这一能力在视觉-语言模型（VLMs）中的组织方式仍不明确，这对模型的鲁棒性有直接影响。我们通过系统的行为和机制分析，探讨了单个物体在遮蔽背景下的上下文推断能力。将VLMs展示单一物体于遮蔽背景中，探测其推断细粒度场景类别和粗粒度上位上下文（室内vs.室外）的能力。发现单个物体在两个层面均能超越偶然推断，性能受同一物体属性调节，这些属性预测人类场景分类。物体身份、场景和上位预测部分解耦：准确推断一个层面不需也不保证准确推断另一层面，且不同模型间的耦合程度差异显著。机制上，当背景上下文被移除时仍保持稳定的物体表示更预测成功上下文推断。场景和上位模式以根本不同的方式扎根：场景身份编码在网络中的图像标记中，而上位信息仅在后期或根本不出现。这些结果揭示了VLMs中上下文推断的组织方式比仅凭准确性所暗示的更复杂，存在行为和机制特征。
英文摘要：
How much scene context a single object carries is a well-studied question in human scene perception, yet how this capacity is organized in vision-language models (VLMs) remains poorly understood, with direct implications for the robustness of these models. We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects. Presenting VLMs with single objects on masked backgrounds, we probe their ability to infer both fine-grained scene category and coarse superordinate context (indoor vs. outdoor). We found that single objects support above-chance inference at both levels, with performance modulated by the same object properties that predict human scene categorization. Object identity, scene, and superordinate predictions are partially dissociable: accurate inference at one level neither requires nor guarantees accurate inference at the others, and the degree of coupling differs markedly across models. Mechanistically, object representations that remain stable when background context is removed are more predictive of successful contextual inference. Scene and superordinate schemas are grounded in fundamentally different ways: scene identity is encoded in image tokens throughout the network, while superordinate information emerges only late or not at all. Together, these results reveal that the organization of contextual inference in VLMs is more complex than accuracy alone suggests, with behavioral and mechanistic signatures
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
