数据高效的对比自监督学习：对监督学习贡献最大的样本对自监督学习贡献最小
Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Least
AI总结：
本文首次证明对比自监督学习中贡献最大的样本是增强最相似的样本，并据此提出数据高效选择方法，可在不影响下游性能下排除20%-40%数据，且这些样本对监督学习贡献最小。
摘要：
自监督学习（SSL）从大量未标记的训练数据中学习高质量的表示。随着数据集变得越来越大，识别出对学习此类表示贡献最大的样本变得至关重要。这可以通过减少所需数据量来实现高效的SSL。然而，量化样本对SSL的价值仍然是一个悬而未决的问题。在这项工作中，我们首次解决了这个问题，通过证明在对比自监督学习中贡献最大的样本是那些在期望上与其他样本具有最相似增强的样本。我们为对比学习在此类子集上的泛化性能提供了严格的保证。通过大量实验，我们表明可以安全地从CIFAR100中排除20%的样本，从STL10和TinyImageNet中排除40%的样本，而不会影响下游任务的性能。总的来说，我们的方法选择的子集在这些数据集上的性能比随机子集高出3%以上。有趣的是，我们还发现对对比学习贡献最大的子集是对监督学习贡献最小的子集。代码可在https://github.com/bigml-cs-ucla/sas-data-efficient-contrastive-learning获取。
英文摘要：
Self-supervised learning (SSL) learns high-quality representations from large pools of unlabeled training data. As datasets grow larger, it becomes crucial to identify the examples that contribute the most to learning such representations. This enables efficient SSL by reducing the volume of data required. Nevertheless, quantifying the value of examples for SSL has remained an open question. In this work, we address this problem for the first time, by proving that examples that contribute the most to contrastive SSL are those that have the most similar augmentations to other examples, in expectation. We provide rigorous guarantees for the generalization performance of contrastive learning on such subsets. Through extensive experiments, we show that we can safely exclude 20% of examples from CIFAR100 and 40% from STL10 and TinyImageNet, without affecting downstream task performance. In general, subsets selected by our method outperform random subsets by over 3% across these datasets. Interestingly, we also discover the subsets that contribute the most to contrastive learning are those that contribute the least to supervised learning. Code available at https://github.com/bigml-cs-ucla/sas-data-efficient-contrastive-learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
