arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.01986cs.LG
收藏

走向民主化的联合嵌入式自监督学习

Towards Democratizing Joint-Embedding Self-Supervised Learning

Mila(米拉研究所);Université de Montréal(蒙特利尔大学);Meta AI

Florian Bordes, Randall Balestriero, Pascal Vincent

摘要:

联合嵌入自监督学习(JE-SSL)近年来发展迅速,因为它可以有效地利用大量未标记的数据。JE-SSL方法的开发主要是通过使用巨大的计算资源来寻求不断提高的下游分类准确性,并且通常建立在从接近的父JE-SSL方法继承的见解和直觉之上。这在不知不觉中导致了许多先入为主的想法,这些想法贯穿于各种方法中,例如Simplified需要非常大的小批量来产生有竞争力的准确性;需要强大而计算缓慢的数据增强。在这项工作中,我们揭穿了几个这样的病态先验思想,希望释放JE-SSL的全部潜力,而不受不必要的限制。事实上,当仔细评估不同下游任务的性能并适当优化方法的超参数时,我们经常(如果不是总是)看到这些普遍存在的误解并不成立。例如,我们表明,它是可能的训练Simplified学习有用的表示,同时使用一个单一的图像补丁作为负面的例子,和简单的高斯噪声作为唯一的数据增强的积极对。沿着这些思路,为了使JE-SSL民主化,并允许研究人员轻松地对其方法进行更广泛的评估,我们引入了一个针对SSL的优化PyTorch库。

英文摘要:

Joint Embedding Self-Supervised Learning (JE-SSL) has seen rapid developments in recent years, due to its promise to effectively leverage large unlabeled data. The development of JE-SSL methods was driven primarily by the search for ever increasing downstream classification accuracies, using huge computational resources, and typically built upon insights and intuitions inherited from a close parent JE-SSL method. This has led unwittingly to numerous pre-conceived ideas that carried over across methods e.g. that SimCLR requires very large mini batches to yield competitive accuracies; that strong and computationally slow data augmentations are required. In this work, we debunk several such ill-formed a priori ideas in the hope to unleash the full potential of JE-SSL free of unnecessary limitations. In fact, when carefully evaluating performances across different downstream tasks and properly optimizing hyper-parameters of the methods, we most often -- if not always -- see that these widespread misconceptions do not hold. For example we show that it is possible to train SimCLR to learn useful representations, while using a single image patch as negative example, and simple Gaussian noise as the only data augmentation for the positive pair. Along these lines, in the hope to democratize JE-SSL and to allow researchers to easily make more extensive evaluations of their methods, we introduce an optimized PyTorch library for SSL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑