对比学习中归纳偏置的理论研究
A Theoretical Study of Inductive Biases in Contrastive Learning
AI总结：
本研究首次从理论上分析了模型架构的归纳偏置对对比学习的影响，证明容量有限的模型会恢复兼容架构的聚类结构而忽略其他结构，并在合成数据上提供了实证支持。
摘要：
理解自监督学习很重要但也充满挑战。先前的理论研究探讨了预训练损失的作用，并将神经网络视为通用的黑盒。然而，Saunshi等人最近的工作指出，模型架构——这一被先前工作很大程度上忽略的组件——也对自监督学习的下游性能有显著影响。在本研究中，我们首次对自监督学习进行了理论分析，纳入了源自模型类的归纳偏置效应。具体而言，我们聚焦于对比学习——一种在视觉领域广泛使用的流行自监督学习方法。我们证明，当模型容量有限时，对比表示会恢复某些与模型架构兼容的特殊聚类结构，而忽略数据分布中的许多其他聚类结构。因此，我们的理论能够捕捉更现实的场景，即对比表示的维度远低于数据分布中聚类的数量。我们在几种合成数据分布上实例化了我们的理论，并提供了支持该理论的实证证据。
英文摘要：
Understanding self-supervised learning is important but challenging. Previous theoretical works study the role of pretraining losses, and view neural networks as general black boxes. However, the recent work of Saunshi et al. argues that the model architecture -- a component largely ignored by previous works -- also has significant influences on the downstream performance of self-supervised learning. In this work, we provide the first theoretical analysis of self-supervised learning that incorporates the effect of inductive biases originating from the model class. In particular, we focus on contrastive learning -- a popular self-supervised learning method that is widely used in the vision domain. We show that when the model has limited capacity, contrastive representations would recover certain special clustering structures that are compatible with the model architecture, but ignore many other clustering structures in the data distribution. As a result, our theory can capture the more realistic setting where contrastive representations have much lower dimensionality than the number of clusters in the data distribution. We instantiate our theory on several synthetic data distributions, and provide empirical evidence to support the theory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
