arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.13808cs.CVcs.AIcs.LG
收藏

自监督学习预训练数据多样性研究

On Pretraining Data Diversity for Self-Supervised Learning

KAUST(阿卜杜拉国王科技大学);University of Oxford(牛津大学)

Hasan Abed Al Kader Hammoud, Tuhin Das, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem

AI总结:

本文研究在固定计算预算下,增加预训练数据多样性对自监督学习性能的影响,发现仅当分布距离极小时提升有效,且分布偏移仍是挑战,实验涵盖七种方法及大规模数据集。

摘要:

我们探讨了在固定计算预算下,使用更多样化的数据集(以唯一样本数量为特征)进行训练对自监督学习(SSL)性能的影响。我们的研究结果一致表明,增加预训练数据多样性可以提升自监督学习性能,但前提是预训练数据与下游数据之间的分布距离极小。值得注意的是,即使通过网络爬取或扩散生成数据等方法实现了极大的预训练数据多样性,分布偏移仍然是一个挑战。我们的实验涵盖了七种自监督学习方法,使用了ImageNet和YFCC100M等大规模数据集,总计超过200个GPU天。代码和训练模型可在 https://github.com/hammoudhasan/DiversitySSL 获取。

英文摘要:

We explore the impact of training with more diverse datasets, characterized by the number of unique samples, on the performance of self-supervised learning (SSL) under a fixed computational budget. Our findings consistently demonstrate that increasing pretraining data diversity enhances SSL performance, albeit only when the distribution distance to the downstream data is minimal. Notably, even with an exceptionally large pretraining data diversity achieved through methods like web crawling or diffusion-generated data, among other ways, the distribution shift remains a challenge. Our experiments are comprehensive with seven SSL methods using large-scale datasets such as ImageNet and YFCC100M amounting to over 200 GPU days. Code and trained models are available at https://github.com/hammoudhasan/DiversitySSL

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑