arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.04470cs.CVcs.AI
收藏

视觉-语言模型中的颜色:CLIP 的缺陷

Color in Visual-Language Models: CLIP deficiencies

Computer Vision Center(计算机视觉中心);Universitat Autònoma de Barcelona(巴塞罗那自治大学)

Guillem Arias, Ramon Baldrich, Maria Vanrell

AI总结:

本文通过合成数据集和 Stroop 测试揭示 CLIP 在颜色编码中存在消色差偏差和文本优先倾向,并从神经元层面分析成因,提出需改进颜色表征机制以增强多模态模型对颜色的理解。

摘要:

本文探讨了颜色在 CLIP(对比语言-图像预训练)中的编码方式,CLIP 是当前人工智能领域最具影响力的视觉-语言模型(VML)。在为本任务创建的合成数据集上进行多项实验后,我们得出结论:CLIP 能够为彩色视觉刺激赋予正确的颜色标签,但我们发现两个主要缺陷:(a) 对与颜色概念关联较弱的消色差刺激存在明显偏差,因此白色、灰色和黑色很少被分配为颜色标签;(b) 倾向于优先处理文本而非其他视觉信息。在此,我们通过一项详尽的 Stroop 效应测试证明,这一倾向在颜色标注中具有高度显著性。为了找出这些颜色缺陷的原因,我们分析了神经元层面的内部表征。我们得出结论:CLIP 中存在大量对文本具有选择性的神经元,尤其是在网络的最深层,而多模态颜色神经元的数量较少,这些神经元可能是正确理解颜色概念的关键。我们的研究强调,有必要改进神经网络中的颜色表征机制,以促进像人类一样更全面地理解颜色,从而提升 CLIP 等多模态模型在现实场景中的有效性和通用性。

英文摘要:

This work explores how color is encoded in CLIP (Contrastive Language-Image Pre-training) which is currently the most influential VML (Visual Language model) in Artificial Intelligence. After performing different experiments on synthetic datasets created for this task, we conclude that CLIP is able to attribute correct color labels to colored visual stimulus, but, we come across two main deficiencies: (a) a clear bias on achromatic stimuli that are poorly related to the color concept, thus white, gray and black are rarely assigned as color labels; and (b) the tendency to prioritize text over other visual information. Here we prove it is highly significant in color labelling through an exhaustive Stroop-effect test. With the aim to find the causes of these color deficiencies, we analyse the internal representation at the neuron level. We conclude that CLIP presents an important amount of neurons selective to text, specially in deepest layers of the network, and a smaller amount of multi-modal color neurons which could be the key of understanding the concept of color properly. Our investigation underscores the necessity of refining color representation mechanisms in neural networks to foster a more comprehensive comprehension of colors as humans understand them, thereby advancing the efficacy and versatility of multimodal models like CLIP in real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑