This work explores how color is encoded in CLIP (Contrastive Language-Image Pre-training) which is currently the most influential VML (Visual Language model) in Artificial Intelligence. After performing different experiments on synthetic datasets created for this task, we conclude that CLIP is able to attribute correct color labels to colored visual stimulus, but, we come across two main deficiencies: (a) a clear bias on achromatic stimuli that are poorly related to the color concept, thus white, gray and black are rarely assigned as color labels; and (b) the tendency to prioritize text over other visual information. Here we prove it is highly significant in color labelling through an exhaustive Stroop-effect test. With the aim to find the causes of these color deficiencies, we analyse the internal representation at the neuron level. We conclude that CLIP presents an important amount of neurons selective to text, specially in deepest layers of the network, and a smaller amount of multi-modal color neurons which could be the key of understanding the concept of color properly. Our investigation underscores the necessity of refining color representation mechanisms in neural networks to foster a more comprehensive comprehension of colors as humans understand them, thereby advancing the efficacy and versatility of multimodal models like CLIP in real-world scenarios.
Image Processing Lab, Universidad de Valencia(图像处理实验室,瓦伦西亚大学);Center for Biomaterials and Tissue Engineering Universitat Politecnica de Valencia(生物材料与组织工程中心,瓦伦西亚理工大学)
Decoding Emotions in Abstract Art: Cognitive Plausibility of CLIP in Recognizing Color-Emotion Associations
Institute for Interdisciplinary Studies, University of Amsterdam(阿姆斯特丹大学跨学科研究所);Department of Information and Computing Sciences, Utrecht University(乌得勒支大学信息与计算科学系)
Laser: Efficient Language-Guided Segmentation in Neural Radiance Fields
Durham University(杜伦大学);Institute of High Performance Computing (IHPC), ASTAR(高性能计算研究所(IHPC),新加坡科技研究局);School of Computer Science, China University of Geosciences(中国地质大学计算机学院);School of Computing, Newcastle University(纽卡斯尔大学计算机学院);University of Chinese Academy of Sciences(中国科学院大学)
RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports
Beijing Institute of Technology(北京理工大学);Beijing Tongren Hospital(北京同仁医院);Capital Medical University(首都医科大学);Beijing Institute of Ophthalmology(北京眼科研究所)
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学研究所)
MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptation
China University of Petroleum-Beijing at Karamay(中国石油大学(北京)克拉玛依校区);Guizhou University(贵州大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);Shanghai Jiao Tong University(上海交通大学)
Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
Academy for Engineering and Technology, Fudan University(复旦大学工程技术研究院);Beijing National Research Center for Information Science and Technology (BNRist)(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
Concept-Guided Prompt Learning for Generalization in Vision-Language Models
Harbin Institute of Technology(哈尔滨工业大学);Southern University of Science and Technology(南方科技大学);Carnegie Mellon University(卡内基梅隆大学);Pengcheng Laboratory(鹏城实验室)
University of Southern California(南加州大学);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学);NVIDIA(英伟达)
对机器人的看似简单的自然语言请求通常都是不明确的,例如“你能把无线鼠标给我吗?候选小鼠的平面图像可能无法提供“无线”所需的区分信息。“世界和其中的物体不是平面图像,而是复杂的3D形状。如果人类根据物体的任何基本属性(如颜色、形状或纹理)请求物体,机器人应该执行必要的探索以完成任务。特别是,虽然在明确理解颜色和类别等视觉属性方面已经取得了实质性的努力和进展,但在理解有关形状和轮廓的语言方面取得的进展相对较少。在这项工作中,我们介绍了一种新的推理任务,目标是关于3D对象的视觉和非视觉语言。我们的新基准ShapeNet Annotated with Referencing Expressions(SNARE)需要一个模型来选择两个对象中的哪一个被自然语言描述引用。我们介绍了几个基于CLIP的模型来区分对象,并表明,虽然最近的进展,联合建模的视觉和语言是有用的机器人语言理解,它仍然是这种情况下,这些基于图像的模型是较弱的理解对象的3D性质-属性,其中发挥了关键作用的操纵。我们发现,将视图估计添加到语言基础模型中可以提高SNARE和在机器人平台上识别语言中提到的对象时的准确性,但请注意,这些模型与人类表现之间存在很大的差距。