通过高斯混合模型理解自监督学习
Understanding Self-Supervised Learning via Gaussian Mixture Models
AI总结：
本文通过高斯混合模型分析自监督学习，证明对比学习能有效找到最优低维子空间，优于传统谱技术，并验证了多模态对比学习的理论结果。
摘要：
自监督学习试图从无标签数据中学习表示；它通过一种损失函数来实现，该函数鼓励一个点的嵌入与其增强版本的嵌入接近。这个简单的想法表现得非常出色，但尚未有明确的理论解释为何如此。在本文中，我们在一个自然的上下文中分析自监督学习：高斯混合模型中的降维。关键在于，我们将数据点的增强定义为从同一底层混合分量中独立抽取的另一个样本。我们证明，普通的对比学习（特别是InfoNCE损失）即使在高斯分布非各向同性的情况下，也能找到最优的低维子空间，而普通的谱技术无法做到这一点。我们还证明了类似的结果适用于“非对比”自监督学习（即SimSiam损失）。我们进一步将分析扩展到多模态对比学习算法（例如CLIP）。在该设置中，我们证明对比学习学习了鱼尔最优子空间的子集，有效地过滤掉了所有噪声。最后，我们通过合成数据实验验证了我们的理论发现。
英文摘要：
Self-supervised learning attempts to learn representations from un-labeled data; it does so via a loss function that encourages the embedding of a point to be close to that of its augmentations. This simple idea performs remarkably well, yet it is not precisely theoretically understood why this is the case. In this paper we analyze self-supervised learning in a natural context: dimensionality reduction in Gaussian Mixture Models. Crucially, we define an augmentation of a data point as being another independent draw from the same underlying mixture component. We show that vanilla contrastive learning (specifically, the InfoNCE loss) is able to find the optimal lower-dimensional subspace even when the Gaussians are not isotropic -- something that vanilla spectral techniques cannot do. We also prove a similar result for "non-contrastive" self-supervised learning (i.e., SimSiam loss). We further extend our analyses to multi-modal contrastive learning algorithms (e.g., CLIP). In this setting we show that contrastive learning learns the subset of fisher-optimal subspace, effectively filtering out all the noise from the learnt representations. Finally, we corroborate our theoretical finding through synthetic data experiments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
