arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.04547cs.LGcs.AI
收藏

CLIP the Bias:数据平衡在多模态学习中有多大作用?

CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?

Google Deepmind(谷歌DeepMind)

Ibrahim Alabdulmohsin, Xiao Wang, Andreas Steiner, Priya Goyal, Alexander D'Amour, Xiaohua Zhai

AI总结:

本研究针对CLIP模型的社会刻板印象偏差问题,提出多模态矩匹配M4算法,分析了数据平衡在多模态学习中的效果、局限与影响因素,给出了提升数据平衡有效性的建议。

摘要:

我们研究了数据平衡在缓解对比语言-图像预训练(CLIP)模型偏差方面的有效性,明确了其优势与局限所在。首先,我们重申了既往结论:CLIP模型会在不经意间吸纳社会刻板印象。为解决这一问题,我们提出了一种名为多模态矩匹配(Multi-Modal Moment Matching,M4)的新算法,旨在降低多模态数据中的表征偏差与关联偏差(即一阶和二阶统计量层面的偏差)。我们借助M4开展了深入分析,考量了模型、表征、数据规模等多种因素。本研究还探究了CLIP学习与消除偏差过程的动态特性。具体而言,我们发现微调能有效对抗表征偏差,但对关联偏差的作用会减弱。此外,数据平衡对模型质量的影响喜忧参半:它往往能提升分类性能,但可能损害检索性能。有趣的是,数据与架构层面的改进似乎能减轻数据平衡对性能的负面影响;例如,将M4应用于带有数据质量过滤器的SigLIP-B/16时,COCO图像到文本的@5检索准确率从(无数据平衡时的)86%提升至87%,ImageNet零样本分类准确率从77%提升至77.5%!最后,我们就提升多模态系统中数据平衡的有效性给出了建议。

英文摘要:

We study the effectiveness of data-balancing for mitigating biases in contrastive language-image pretraining (CLIP), identifying areas of strength and limitation. First, we reaffirm prior conclusions that CLIP models can inadvertently absorb societal stereotypes. To counter this, we present a novel algorithm, called Multi-Modal Moment Matching (M4), designed to reduce both representation and association biases (i.e. in first- and second-order statistics) in multimodal data. We use M4 to conduct an in-depth analysis taking into account various factors, such as the model, representation, and data size. Our study also explores the dynamic nature of how CLIP learns and unlearns biases. In particular, we find that fine-tuning is effective in countering representation biases, though its impact diminishes for association biases. Also, data balancing has a mixed impact on quality: it tends to improve classification but can hurt retrieval. Interestingly, data and architectural improvements seem to mitigate the negative impact of data balancing on performance; e.g. applying M4 to SigLIP-B/16 with data quality filters improves COCO image-to-text retrieval @5 from 86% (without data balancing) to 87% and ImageNet 0-shot classification from 77% to 77.5%! Finally, we conclude with recommendations for improving the efficacy of data balancing in multimodal systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑