用于多模态对比学习的CLIPLoss与基于范数的数据选择方法
CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning
AI总结：
针对CLIP预训练数据选择，提出改进模态对齐的s-CLIPLoss和基于范数的NormSim方法，在DataComp基准上显著提升性能并达到新SOTA。
摘要：
数据选择已成为大规模视觉语言模型预训练（如CLIP）的核心问题，尤其是在存在噪声的网络整理数据集的情况下。三种主要的数据选择方法是：（1）利用外部非CLIP模型辅助数据选择；（2）训练新的CLIP风格嵌入模型，使其在选择高质量数据方面比原始OpenAI CLIP模型更有效；（3）设计普遍适用于任何CLIP嵌入的更好指标或策略，而无需特定模型属性（例如，CLIPScore是一种流行指标）。尽管前两种方法已被广泛研究，但第三种方法仍有待深入探索。本文通过提出两种新方法推进了第三种方法。首先，不同于仅考虑单个样本中两种模态之间对齐的经典CLIP分数，我们引入了surrogate-CLIPLoss（s-CLIPLoss），这是一种受CLIP损失启发的方法，将一个样本与其对比对之间的对齐作为额外的归一化项，以实现更好的质量度量。其次，当已知下游任务时，我们提出了一种新的基于范数的指标NormSim，用于衡量预训练数据与目标数据之间的相似度。我们在数据选择基准DataComp上测试了我们的方法。与仅使用OpenAI CLIP-L/14的最佳基线相比，我们的方法在ImageNet-1k上实现了5.3%的提升，在38个下游评估任务上实现了2.8%的提升。此外，s-CLIPLoss和NormSim均与现有技术兼容。通过将我们的方法与当前最佳方法DFN和HYPE结合，可将下游任务的平均性能提升0.9%，在DataComp-medium基准上达到新的最先进水平。
英文摘要：
Data selection has emerged as a core issue for large-scale visual-language model pretaining (e.g., CLIP), particularly with noisy web-curated datasets. Three main data selection approaches are: (1) leveraging external non-CLIP models to aid data selection, (2) training new CLIP-style embedding models that are more effective at selecting high-quality data than the original OpenAI CLIP model, and (3) designing better metrics or strategies universally applicable to any CLIP embedding without requiring specific model properties (e.g., CLIPScore is one popular metric). While the first two approaches have been extensively studied, the third remains under-explored. In this paper, we advance the third approach by proposing two new methods. Firstly, instead of classical CLIP scores that only consider the alignment between two modalities from a single sample, we introduce surrogate-CLIPLoss (s-CLIPLoss), a CLIP loss-inspired method that adds the alignment between one sample and its contrastive pairs as an extra normalization term for better quality measurement. Secondly, when downstream tasks are known, we propose a new norm-based metric, NormSim, to measure the similarity between pretraining data and target data. We test our methods on the data selection benchmark, DataComp~\cite{gadre2023datacomp}. Compared to the best baseline using only OpenAI's CLIP-L/14, our methods achieve a 5.3\% improvement on ImageNet-1k and a 2.8\% improvement on 38 downstream evaluation tasks. Moreover, both s-CLIPLoss and NormSim are compatible with existing techniques. By combining our methods with the current best methods DFN and HYPE, we can boost average performance on downstream tasks by 0.9\%, achieving a new state-of-the-art on the DataComp-medium benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
