利用图像描述改进多模态数据集
Improving Multimodal Datasets with Image Captioning
AI总结：
本文研究利用图像描述模型生成合成描述以提升网络抓取多模态数据质量，通过混合原始与生成描述的策略在DataComp基准上超越最佳过滤方法，并分析了合成文本的效用与局限性。
摘要：
大规模网络数据集在CLIP和Flamingo等大型视觉-语言模型的成功中发挥着关键作用。然而，原始网络数据存在噪声，而现有的降噪过滤方法往往以牺牲数据多样性为代价。我们的工作聚焦于描述质量这一主要噪声来源，并研究生成的描述如何提高带有不明确文本的网络抓取数据点的效用。通过探索原始描述与生成描述的不同混合策略，在给定128M图像-文本对候选池的情况下，我们在ImageNet上比DataComp基准提出的最佳过滤方法高出2%，在38个任务上平均高出4%。我们的最佳方法在Flickr和MS-COCO检索上的表现也提高了2倍。随后，我们分析了合成描述成为有效文本监督来源的原因。在对不同图像描述模型进行实验时，我们还证明模型在标准图像描述基准（如NoCaps CIDEr）上的性能并不能可靠地指示其生成的描述在多模态训练中的效用。最后，我们在DataComp大规模（1.28B图像-文本对）上使用生成描述的实验揭示了合成文本的局限性，以及随着训练数据量增加图像筛选的重要性。我们实验中使用的合成描述现已在HuggingFace上公开。
英文摘要：
Massive web datasets play a key role in the success of large vision-language models like CLIP and Flamingo. However, the raw web data is noisy, and existing filtering methods to reduce noise often come at the expense of data diversity. Our work focuses on caption quality as one major source of noise, and studies how generated captions can increase the utility of web-scraped datapoints with nondescript text. Through exploring different mixing strategies for raw and generated captions, we outperform the best filtering method proposed by the DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a candidate pool of 128M image-text pairs. Our best approach is also 2x better at Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an effective source of text supervision. In experimenting with different image captioning models, we also demonstrate that the performance of a model on standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable indicator of the utility of the captions it generates for multimodal training. Finally, our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text, as well as the importance of image curation with increasing training data quantity. The synthetic captions used in our experiments are now available on HuggingFace.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
