我们研究了数据平衡在缓解对比语言-图像预训练(CLIP)模型偏差方面的有效性,明确了其优势与局限所在。首先,我们重申了既往结论:CLIP模型会在不经意间吸纳社会刻板印象。为解决这一问题,我们提出了一种名为多模态矩匹配(Multi-Modal Moment Matching,M4)的新算法,旨在降低多模态数据中的表征偏差与关联偏差(即一阶和二阶统计量层面的偏差)。我们借助M4开展了深入分析,考量了模型、表征、数据规模等多种因素。本研究还探究了CLIP学习与消除偏差过程的动态特性。具体而言,我们发现微调能有效对抗表征偏差,但对关联偏差的作用会减弱。此外,数据平衡对模型质量的影响喜忧参半:它往往能提升分类性能,但可能损害检索性能。有趣的是,数据与架构层面的改进似乎能减轻数据平衡对性能的负面影响;例如,将M4应用于带有数据质量过滤器的SigLIP-B/16时,COCO图像到文本的@5检索准确率从(无数据平衡时的)86%提升至87%,ImageNet零样本分类准确率从77%提升至77.5%!最后,我们就提升多模态系统中数据平衡的有效性给出了建议。
英文摘要:
We study the effectiveness of data-balancing for mitigating biases in contrastive language-image pretraining (CLIP), identifying areas of strength and limitation. First, we reaffirm prior conclusions that CLIP models can inadvertently absorb societal stereotypes. To counter this, we present a novel algorithm, called Multi-Modal Moment Matching (M4), designed to reduce both representation and association biases (i.e. in first- and second-order statistics) in multimodal data. We use M4 to conduct an in-depth analysis taking into account various factors, such as the model, representation, and data size. Our study also explores the dynamic nature of how CLIP learns and unlearns biases. In particular, we find that fine-tuning is effective in countering representation biases, though its impact diminishes for association biases. Also, data balancing has a mixed impact on quality: it tends to improve classification but can hurt retrieval. Interestingly, data and architectural improvements seem to mitigate the negative impact of data balancing on performance; e.g. applying M4 to SigLIP-B/16 with data quality filters improves COCO image-to-text retrieval @5 from 86% (without data balancing) to 87% and ImageNet 0-shot classification from 77% to 77.5%! Finally, we conclude with recommendations for improving the efficacy of data balancing in multimodal systems.
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Department of Computer Science and Engineering(计算机科学与工程系);Incheon National University(庆尚国立大学);Department of Artificial Intelligence(人工智能系);Inha University(Inha大学);Center for Brain-Machine Interface(脑机接口中心)
Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features
Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
仇恨模因是社交媒体上日益增长的威胁。虽然模因中的图像及其对应的文本是相关的,但当单独查看时,它们不一定传达相同的含义。因此,检测仇恨模因需要仔细考虑视觉和文本信息。多模态预训练对于这项任务是有益的,因为它通过在相似的特征空间中表示图像和文本来有效地捕捉它们之间的关系。此外,它是必要的,通过中间融合的图像和文本特征之间的相互作用建模。大多数现有的方法要么采用多模式预训练,要么采用中间融合,但不是两者兼而有之。在这项工作中,我们提出了Hate-CLIPper架构,该架构通过一个特征交互矩阵(CLIP)明确地对使用对比图像预训练(CLIP)编码器获得的图像和文本表示之间的跨模态交互进行建模。一个简单的分类器,基于Hateful Memes Challenge(HMC)数据集,能够达到最先进的性能,AUROC为85.8,甚至超过了人类的82.65。在Propaganda Memes和TamilMemes等模因数据集上的实验也证明了该方法的通用性。最后,我们分析的可解释性的跨模态表示,并表明,跨模态的相互作用确实可以促进有意义的概念的学习。这个工作的代码可以在https://github.com/gokulkarthik/hateclipper上找到。
Balancing Multimodal Training Through Game-Theoretic Regularization
Department of Electrical Engineering, KU Leuven(KU Leuven 电子工程系);Department of Development and Regeneration, KU Leuven(KU Leuven 发展与再生系);Media Lab and EECS, MIT(MIT 媒体实验室和电子工程与计算机科学系)
Multimodal neural networks better explain multivoxel patterns in the hippocampus
CerCo CNRS, UMR 5549(CerCo 法国国家科学研究中心联合实验室(UMR 5549));Université de Toulouse(图卢兹大学);IRIT CNRS, UMR 5505(IRIT 法国国家科学研究中心联合实验室(UMR 5505));ANITI(ANITI(图卢兹大学))
人类海马体拥有“概念细胞”,当受到属于特定概念的刺激时,神经元会放电,而不管其形式如何。最近,在称为CLIP的多模态网络中发现了类似的概念细胞(Radford et al.,2021年)。在这里,我们问CLIP是否能比纯粹的视觉(或语言)模型更好地解释人类海马体的fMRI活动。我们将我们的分析扩展到一系列公开的单模态和多模态模型。我们证明,“多模态”突出作为一个关键组成部分时,评估网络的能力来解释海马体中的多体素活动。
Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks
Max-Planck-Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(蒂宾根人工智能中心);University of Tübingen(蒂宾根大学);ELLIS Institute Tübingen(ELLIS蒂宾根研究所)
Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusion
Department of Informatics University of Bern(伯尔尼大学信息学院);College of Computing and Data Science Nanyang Technological University(南洋理工大学计算机与数据科学学院);School of Software Engineering Xi’an Jiaotong University(西安交通大学软件工程学院)