Sieve：利用图像字幕模型进行多模态数据集剪枝
Sieve: Multimodal Dataset Pruning Using Image Captioning Models
AI总结：
针对现有CLIPScore剪枝方法的假阳性/阴性问题，提出基于图像字幕模型合成字幕的剪枝信号Sieve，结合语言模型嵌入空间的语义相似度评估图文对齐度，在DataComp基准的下游任务和检索任务上均优于CLIPScore。
摘要：
视觉语言模型（Vision-Language Models, VLMs）在大规模、多样化且存在噪声的网页爬取数据集上进行预训练，这凸显了数据集剪枝的关键必要性——因为这类数据集的质量与VLM在下游任务上的表现强相关。利用预训练模型输出的CLIPScore筛选高对齐度样本用于模型训练，是目前最成功的剪枝方法之一。我们认为该方法存在多重局限，包括CLIP在噪声标签上预训练导致的假阳性与假阴性问题。\n我们提出一种剪枝信号Sieve，它利用在小规模、多样化且对齐良好的图文对上预训练的图像字幕模型生成的合成字幕，来评估含噪声图文对的对齐程度。为弥补生成字幕多样性有限与替代文本（alt-text）高度多样性之间的差距，我们在基于无标注文本语料预训练的语言模型的嵌入空间中，估计文本语义相似度。\n在多模态数据集过滤基准DataComp上，针对38个下游任务评估时，我们的剪枝方法在中等规模和大规模设置下分别比CLIPScore高出2.6%和1.7%。此外，在检索任务上，Sieve在中等规模和大规模设置下分别带来2.7%和4.5%的显著提升。
英文摘要：
Vision-Language Models (VLMs) are pretrained on large, diverse, and noisy web-crawled datasets. This underscores the critical need for dataset pruning, as the quality of these datasets is strongly correlated with the performance of VLMs on downstream tasks. Using CLIPScore from a pretrained model to only train models using highly-aligned samples is one of the most successful methods for pruning. We argue that this approach suffers from multiple limitations including: false positives and negatives due to CLIP's pretraining on noisy labels. We propose a pruning signal, Sieve, that employs synthetic captions generated by image-captioning models pretrained on small, diverse, and well-aligned image-text pairs to evaluate the alignment of noisy image-text pairs. To bridge the gap between the limited diversity of generated captions and the high diversity of alternative text (alt-text), we estimate the semantic textual similarity in the embedding space of a language model pretrained on unlabeled text corpus. Using DataComp, a multimodal dataset filtering benchmark, when evaluating on 38 downstream tasks, our pruning approach, surpasses CLIPScore by 2.6\% and 1.7\% on medium and large scale respectively. In addition, on retrieval tasks, Sieve leads to a significant improvement of 2.7% and 4.5% on medium and large scale respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
