VeCLIP：通过视觉增强字幕改进CLIP训练
VeCLIP: Improving CLIP Training via Visual-enriched Captions
AI总结：
本文提出VeCLIP，通过引入视觉增强字幕（VeCap）的可扩展改写流程和混合训练方案，在大规模网络数据上提升CLIP的图像-文本对齐与模型性能，显著优于基线。
摘要：
大规模网络爬取的数据集是预训练视觉-语言模型（如CLIP）成功的基础。然而，网络爬取的AltTexts固有的噪声和潜在的不相关性给实现精确的图像-文本对齐带来了挑战。现有利用大型语言模型（LLMs）进行字幕改写的方法在CC3M和CC12M等小型精选数据集上显示出前景。本研究引入了一种可扩展的噪声字幕改写流程。与近期LLM改写技术不同，我们强调将视觉概念融入字幕中，称为视觉增强字幕（VeCap）。为确保数据多样性，我们提出了一种新颖的混合训练方案，优化了AltTexts与新生成的VeCap的联合利用。我们展示了该方法在大规模网络爬取数据集上训练CLIP的适应性，称为VeCLIP。采用这种成本效益高的流程，我们轻松地将数据集扩展到3亿个样本，命名为VeCap数据集。我们的结果显示在图像-文本对齐和整体模型性能方面具有显著优势。例如，在12M设置下，VeCLIP在COCO和Flickr30k检索任务上实现了高达+25.2%的提升。在数据效率方面，VeCLIP仅使用原始CLIP所用数据的14%和ALIGN的11%就实现了+3%的提升。我们还注意到VeCap数据与其他精选的适合零样本分类任务的数据集具有互补性。当结合VeCap和DFN时，我们的模型在图像-文本检索和零样本分类任务上均能取得强劲性能，例如，H/14模型在ImageNet零样本上达到83.1%的准确率@1。我们在https://github.com/apple/ml-veclip发布了预训练模型。
英文摘要：
Large-scale web-crawled datasets are fundamental for the success of pre-training vision-language models, such as CLIP. However, the inherent noise and potential irrelevance of web-crawled AltTexts pose challenges in achieving precise image-text alignment. Existing methods utilizing large language models (LLMs) for caption rewriting have shown promise on small, curated datasets like CC3M and CC12M. This study introduces a scalable pipeline for noisy caption rewriting. Unlike recent LLM rewriting techniques, we emphasize the incorporation of visual concepts into captions, termed as Visual-enriched Captions (VeCap). To ensure data diversity, we propose a novel mixed training scheme that optimizes the utilization of AltTexts alongside newly generated VeCap. We showcase the adaptation of this method for training CLIP on large-scale web-crawled datasets, termed VeCLIP. Employing this cost-effective pipeline, we effortlessly scale our dataset up to 300 million samples named VeCap dataset. Our results show significant advantages in image-text alignment and overall model performance. For example, VeCLIP achieves up to +25.2% gain in COCO and Flickr30k retrieval tasks under the 12M setting. For data efficiency, VeCLIP achieves +3% gain while only using 14% of the data employed in the vanilla CLIP and 11% in ALIGN. We also note the VeCap data is complementary with other well curated datasets good for zero-shot classification tasks. When combining VeCap and DFN, our model can achieve strong performance on both of image-text retrieval and zero-shot classification tasks, e.g. 83.1% accuracy@1 on ImageNet zero-shot for a H/14 model. We release the pre-trained models at https://github.com/apple/ml-veclip.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
