CLIP 表现得像一个词袋模型但不是单模态的
CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
AI总结：
CLIP表现出词袋模型特性，但其属性-对象绑定信息已单模态编码，通过简单线性变换可提升性能。
摘要：
CLIP（对比语言-图像预训练）已成为各种下游任务中流行的选择。然而，最近的研究对其有效表示组合概念的能力提出了质疑。这些研究指出，CLIP往往像词袋模型（BoW）一样运作，将图像和文本解释为独立概念的集合，而没有把握结构关系。特别是，CLIP在图像或文本中存在多个对象时，难以正确将属性绑定到相应对象。在本工作中，我们研究了CLIP为何表现出这种BoW-like行为。我们的关键发现是CLIP并不缺乏绑定信息。通过线性探测、增加对象数量的鲁棒性测试以及联合搜索实验，我们显示属性-对象绑定已编码在CLIP的文本和图像嵌入中。弱点在于跨模态对齐，它未能保留此信息。我们显示可以通过简单的线性变换到文本嵌入来跨模态访问此信息。这提高了CLIP的属性-对象绑定性能，并证实信息已单模态编码。在实践中，这意味着基于CLIP的系统可以通过在现有嵌入上训练的轻量线性层进行增强，避免昂贵的编码器重新训练。代码可在https://github.com/kdariina/CLIP-not-BoW-unimodally获取。
英文摘要：
CLIP (Contrastive Language-Image Pretraining) has become a popular choice for various downstream tasks. However, recent studies have questioned its ability to represent compositional concepts effectively. These works suggest that CLIP often acts like a bag-of-words (BoW) model, interpreting images and text as sets of individual concepts without grasping the structural relationships. In particular, CLIP struggles to correctly bind attributes to their corresponding objects when multiple objects are present in an image or text. In this work, we investigate why CLIP exhibits this BoW-like behavior. Our key finding is that CLIP does not lack binding information. Through linear probing, robustness tests with increasing object counts, and conjunctive search experiments, we show that attribute-object bindings are already encoded within CLIP's text and image embeddings. The weakness lies in the cross-modal alignment, which fails to preserve this information. We show it can be accessed cross-modally with a simple linear transformation to text embeddings. This improves CLIP's attribute-object binding performance and confirms that the information was already encoded unimodally. In practice, this means CLIP-based systems can be enhanced with a lightweight linear layer trained on existing embeddings, avoiding costly encoder retraining. The code is available at https://github.com/kdariina/CLIP-not-BoW-unimodally.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
