对比表示学习的统计一致性与泛化性
Statistical Consistency and Generalization of Contrastive Representation Learning
AI总结：
本文提出统一的统计学习理论，证明对比损失与最优排序统计一致，并推导出随负样本数增加而改善的泛化界，解释了大负样本集的经验优势。
摘要：
对比表示学习（CRL）支撑着许多现代基础模型。尽管最近取得了理论进展，现有分析仍存在几个关键限制：（i）CRL的统计一致性仍知之甚少；（ii）可用的泛化界随着负样本数量的增加而恶化，这与大负样本集的经验优势相矛盾；（iii）CRL的检索性能受到的理论关注有限。在本文中，我们为CRL发展了一个统一的统计学习理论。对于下游任务，我们使用AUC型总体准则评估检索质量，并证明对比损失与最优排序是 extit{统计一致的}。我们进一步建立了一个 extit{校准型不等式}，定量地将过剩对比风险与过剩检索次优性联系起来。对于上游训练，我们研究了监督和自监督对比目标，并分别推导了阶为$O(1/m + 1/\sqrt{n})$和$O(1/\sqrt{m} + 1/\sqrt{n})$的泛化界，其中$m$表示负样本数量，$n$表示锚点数量。这些界不仅解释了大负样本集的经验优势，还揭示了$m$和$n$之间的显式权衡。在大规模视觉-语言模型上的广泛实验证实了我们的理论预测。
英文摘要：
Contrastive representation learning (CRL) underpins many modern foundation models. Despite recent theoretical progress, existing analyses suffer from several key limitations: (i) the statistical consistency of CRL remains poorly understood; (ii) available generalization bounds deteriorate as the number of negative samples increases, contradicting the empirical benefits of large negative sets; and (iii) the retrieval performance of CRL has received limited theoretical attention. In this paper, we develop a unified statistical learning theory for CRL. For downstream tasks, we evaluate retrieval quality using an AUC-type population criterion and show that the contrastive loss is \emph{statistically consistent} with optimal ranking. We further establish a \emph{calibration-style inequality} that quantitatively relates excess contrastive risk to excess retrieval suboptimality. For upstream training, we study both supervised and self-supervised contrastive objectives and derive generalization bounds of order $O(1/m + 1/\sqrt{n})$ and $O(1/\sqrt{m} + 1/\sqrt{n})$, respectively, where $m$ denotes the number of negative samples and $n$ the number of anchor points. These bounds not only explain the empirical advantages of large negative sets but also reveal an explicit trade-off between $m$ and $n$. Extensive experiments on large-scale vision--language models corroborate our theoretical predictions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
