关于对比与非对比自监督学习的对偶性
On the duality between contrastive and non-contrastive self-supervised learning
AI总结：
本文研究对比与非对比自监督学习的理论相似性，证明两类方法在有限假设下等价，并通过超参数调优和设计选择缩小性能差距，推动统一SOTA方法以深化理解。
摘要：
近期图像表示的自监督学习方法可分为不同的方法族，特别是对比和非对比方法。虽然两族之间的差异已被充分讨论以激发新方法，但我们更关注它们之间的理论相似性。通过设计可在代数上相关联且在有限假设下被证明等价的对比和基于协方差的非对比准则，我们展示了这些方法族之间可以有多接近。我们进一步研究流行方法并引入其变体，使我们能将此理论结果与当前实践联系起来，并展示设计选择对下游性能的影响（或无影响）。受等价结果启发，我们调查了SimCLR的低性能问题，并展示其如何通过仔细的超参数调优匹配VICReg的性能，显著超越已知基线。我们还挑战了非对比方法需要大输出维度的流行假设。我们的理论和定量结果表明，在特定情况下，对比与非对比方法之间的数值差距可通过更好的网络设计选择和超参数调优来缩小。证据显示，统一不同SOTA方法是建立对自监督学习更好理解的重要方向。
英文摘要：
Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While differences between the two families have been thoroughly discussed to motivate new approaches, we focus more on the theoretical similarities between them. By designing contrastive and covariance based non-contrastive criteria that can be related algebraically and shown to be equivalent under limited assumptions, we show how close those families can be. We further study popular methods and introduce variations of them, allowing us to relate this theoretical result to current practices and show the influence (or lack thereof) of design choices on downstream performance. Motivated by our equivalence result, we investigate the low performance of SimCLR and show how it can match VICReg's with careful hyperparameter tuning, improving significantly over known baselines. We also challenge the popular assumption that non-contrastive methods need large output dimensions. Our theoretical and quantitative results suggest that the numerical gaps between contrastive and non-contrastive methods in certain regimes can be closed given better network design choices and hyperparameter tuning. The evidence shows that unifying different SOTA methods is an important direction to build a better understanding of self-supervised learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
