带间隔的对比学习的 VC 维最优界
Optimal VC Dimension of Contrastive Learning with Margin
浏览论文内容
中文总结 AI 辅助
本文解决了带间隔的对比学习的 VC 维问题，证明了其 VC 维为 $O(n/\alpha^2)$，并给出匹配下界，从而确定了最优界。
中文摘要 AI 辅助
对比学习是一种成功的范式，用于从一组“锚点-正样本-负样本”三元组 $(i,j^{+},k^{-})$ 中学习 $d$ 维几何表示，其中三元组表示“项目 $i$ 比 $k$ 更接近 $j$”。尽管取得了成功，但理解为什么对比学习能够产生高泛化质量的表示——超越 PAC 学习通常悲观的预测——仍然是一个核心问题。最近，\n\citet*{alon2024optimal} 证明了，对于 $n$ 点数据集的 $d$ 维欧几里得表示的 PAC 学习，$\n\Theta(\min(nd, n^2))$ 个三元组是必要且充分的，同时他们提出了一个开放性问题：他们针对更现实的“带间隔的对比学习”场景的 VC 维界是否可以改进。对于间隔参数 $\alpha > 0$，如果嵌入 $\phi:[n]\rightarrow \mathbb{R}^{d}$ 满足 $\\|\phi(i)-\phi(k)\\|_2>(1+\alpha)\cdot\\|\phi(i)-\phi(j)\\|_2$，则三元组 $(i,j^{+},k^{-})_{\alpha}$ 被满足。在这项工作中，我们解决了他们的问题，证明了在任何间隔 $\alpha\in(0,1)$ 下，对比学习的 VC 维实际上是 $O(n/\alpha^2)$，改进了之前 $O(n\log(n)/\alpha^2)$ 的界。我们还通过提供匹配的下界 $\Omega(\frac{n}{\alpha^2})$（之前已知的下界为 $\Omega(\frac{n}{\alpha})$），证明了这些界在常数因子内是最优的，适用于 $\alpha\geq \max(n^{-1/2},d^{-1/2})$。
英文摘要
Contrastive learning is a successful paradigm for learning $d$-dimensional geometric representations from a collection of ``anchor--positive--negative'' triplets $(i,j^{+},k^{-})$, indicating that ``item $i$ is closer to $j$ than to $k$.'' Despite its success, understanding why contrastive learning leads to representations of high \textit{generalization} quality---beyond the often pessimistic predictions from PAC-learning---remains a central question. Recently, \citet*{alon2024optimal} proved that, for PAC-learning $d$-dimensional Euclidean representations of $n$-point datasets, $Θ(\min(nd, n^2))$ triplets are necessary and sufficient, while they posed as an open question whether their VC dimension bounds for the more realistic setting of \textit{contrastive learning with a margin} can be improved. For a margin parameter $α>0$, a triplet $(i,j^{+},k^{-})_α$ is satisfied by the embedding $ϕ:[n]\rightarrow \mathbb{R}^{d}$, if $\|ϕ(i)-ϕ(k)\|_2>(1+α)\cdot\|ϕ(i)-ϕ(j)\|_2$. In this work, we resolve their question by proving that the VC dimension of contrastive learning under any margin $α\in(0,1)$ is in fact $O(n/α^2)$, improving on the previous bound of $O(n\log(n)/α^2)$. We also establish that the bounds are optimal up to constant factors, by providing a matching lower bound of $Ω(\frac{n}{α^2})$ (the previously known lower bound was $Ω(\frac{n}α)$), for $α\geq \max(n^{-1/2},d^{-1/2})$.
发表机构
- Northwestern University（西北大学）
- UC Santa Cruz（加州大学圣克鲁兹分校）
机构由 AI 辅助整理，请以论文原文为准。
