预训练数据在迁移学习中的作用
The Role of Pre-training Data in Transfer Learning
摘要：
模型预训练和后续微调的迁移学习范例产生高精度模型。虽然大多数研究都建议扩大预训练的规模，以便从迁移学习中获益最多，但问题仍然存在：应该使用什么数据和方法进行预训练？我们使用3种预训练方法（监督、对比语言-图像和图像-图像）、7个预训练数据集和9个下游数据集研究了预训练数据分布对Few-Shot和完全微调性能的影响。通过广泛的控制实验，我们发现预训练数据源的选择对于Few-Shot传输是必不可少的，但是随着更多的数据可用于微调，其作用降低。此外，我们还探索了数据整理的作用，并研究了标签噪声和预训练数据集大小之间的权衡。我们发现，使用2000倍的LAION预训练数据可以匹配监督ImageNet预训练的性能。此外，我们研究了预训练方法的效果，比较了语言-图像对比与图像-图像对比，发现后者导致更好的下游准确性
英文摘要：
The transfer learning paradigm of model pre-training and subsequent fine-tuning produces high-accuracy models. While most studies recommend scaling the pre-training size to benefit most from transfer learning, a question remains: what data and method should be used for pre-training? We investigate the impact of pre-training data distribution on the few-shot and full fine-tuning performance using 3 pre-training methods (supervised, contrastive language-image and image-image), 7 pre-training datasets, and 9 downstream datasets. Through extensive controlled experiments, we find that the choice of the pre-training data source is essential for the few-shot transfer, but its role decreases as more data is made available for fine-tuning. Additionally, we explore the role of data curation and examine the trade-offs between label noise and the size of the pre-training dataset. We find that using 2000X more pre-training data from LAION can match the performance of supervised ImageNet pre-training. Furthermore, we investigate the effect of pre-training methods, comparing language-image contrastive vs. image-image contrastive, and find that the latter leads to better downstream accuracy
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
