InfoNCE 诱导高斯分布
InfoNCE Induces Gaussian Distribution
AI总结：
本文证明对比学习中的 InfoNCE 损失函数会使表示学习产生高斯分布结构，通过理论分析和实验验证了这种高斯行为。
摘要：
对比学习已成为现代表示学习的基石，允许使用大量无标签数据训练任务特定和通用（基础）模型。对比训练中的典型损失是 InfoNCE 及其变体。在这项工作中，我们展示了 InfoNCE 目标在对比训练中诱导出表示的高斯结构。我们在两个互补的机制中建立了这一结果。首先，我们表明在某些对齐和集中假设下，高维表示的投影渐近地接近多元高斯分布。其次，在较不严格的假设下，我们表明添加一个促进低特征范数和高特征熵的小渐近消失正则化项会导致类似的渐近结果。我们通过在合成和 CIFAR-10 数据集上使用多种编码器架构和大小的实验支持我们的分析，展示了一致的高斯行为。这一视角为对比表示中常见的高斯性提供了原则性解释。由此产生的高斯模型使得对学习表示进行原则性分析处理成为可能，并有望支持对比学习中的广泛应用。
英文摘要：
Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models. A prototypical loss in contrastive training is InfoNCE and its variants. In this work, we show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training. We establish this result in two complementary regimes. First, we show that under certain alignment and concentration assumptions, projections of the high-dimensional representation asymptotically approach a multivariate Gaussian distribution. Next, under less strict assumptions, we show that adding a small asymptotically vanishing regularization term that promotes low feature norm and high feature entropy leads to similar asymptotic results. We support our analysis with experiments on synthetic and CIFAR-10 datasets across multiple encoder architectures and sizes, demonstrating consistent Gaussian behavior. This perspective provides a principled explanation for commonly observed Gaussianity in contrastive representations. The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
