arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.20087cs.CV
收藏

Too Large;面向视觉-语言预训练的数据缩减

Too Large; Data Reduction for Vision-Language Pre-Training

National University of Singapore(新加坡国立大学)

Alex Jinpeng Wang, Kevin Qinghong Lin, David Junhao Zhang, Stan Weixian Lei, Mike Zheng Shou

AI总结:

针对大规模 VLP 数据的图文错位与冗余问题,本文提出 TL;DR,通过代表性样本选择和描述重写实现高比例数据压缩,并在七个下游任务上保持或提升模型性能。

摘要:

本文考察了广泛使用的大规模视觉-语言预训练(VLP)数据集中存在的严重图文错位和高冗余问题。为解决这些问题,我们提出了一种高效且直接的视觉-语言学习算法 TL;DR,旨在将现有的大规模 VLP 数据压缩为一个小型、高质量的集合。我们的方法包括两个主要步骤。首先,开发一个基于码本的编码器-解码器描述生成器,用于选择具有代表性的样本。其次,为所选样本生成新的描述,以补充原始描述,在保持独特性的同时缓解文本-图像错位问题。因此,TL;DR 使我们能够将大型数据集缩减为一小组高质量数据,这些数据可作为替代预训练数据集。该算法显著加快了耗时的预训练过程。具体而言,TL;DR 能够以高压缩比压缩主流 VLP 数据集,例如,将经过良好清洗的 CC3M 数据集从 2.82M 缩减至 0.67M(约24%),并将含噪的 YFCC15M 从 15M 缩减至 2.5M(约16.7%)。在七个下游任务上使用三种流行 VLP 模型进行的大量实验表明,与在全规模数据集上训练相比,在 TL;DR 提供的压缩数据集上训练的 VLP 模型能够取得相近甚至更好的结果。代码将在 https://github.com/showlab/datacentric.vlp 公开。

英文摘要:

This paper examines the problems of severe image-text misalignment and high redundancy in the widely-used large-scale Vision-Language Pre-Training (VLP) datasets. To address these issues, we propose an efficient and straightforward Vision-Language learning algorithm called TL;DR, which aims to compress the existing large VLP data into a small, high-quality set. Our approach consists of two major steps. First, a codebook-based encoder-decoder captioner is developed to select representative samples. Second, a new caption is generated to complement the original captions for selected samples, mitigating the text-image misalignment problem while maintaining uniqueness. As the result, TL;DR enables us to reduce the large dataset into a small set of high-quality data, which can serve as an alternative pre-training dataset. This algorithm significantly speeds up the time-consuming pretraining process. Specifically, TL;DR can compress the mainstream VLP datasets at a high ratio, e.g., reduce well-cleaned CC3M dataset from 2.82M to 0.67M ($\sim$24\%) and noisy YFCC15M from 15M to 2.5M ($\sim$16.7\%). Extensive experiments with three popular VLP models over seven downstream tasks show that VLP model trained on the compressed dataset provided by TL;DR can perform similar or even better results compared with training on the full-scale dataset. The code will be made available at \url{https://github.com/showlab/datacentric.vlp}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑