草图线性对比学习：近似、优化与统计缩放
Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scaling
AI总结：
研究对比学习在草图线性模型下的缩放规律，通过分析高斯潜变量配对设置，推导出风险分解和显式缩放定律，揭示了优化与有限样本噪声随模型大小、数据和训练时间的变化。
摘要：
缩放定律描述了学习性能如何随模型大小、数据大小和计算资源变化。虽然近期理论工作已为草图线性回归建立了缩放定律，但对对比表示学习的理解仍较少。本文研究配对高斯潜变量设置下对比学习的草图线性模型。学习器仅观测两个相关变量的草图视图，并通过全批量经验梯度下降训练双线性对比得分。我们在对齐幂律谱和对比源条件下分析高斯负二次对比代理，将风险分解为不可约风险、近似误差、GD偏差、GD方差和交叉项。交叉项受偏差和方差控制，因此不影响上界缩放。我们的主要定理给出了关于草图维度$M$、样本量$N$和有效优化视界$L_{\mathrm{eff}}\gamma$的显式缩放定律。与标准线性回归缩放定律相比，对比设置必须学习两个视图之间的交互，这改变了优化和有限样本噪声随模型大小、数据和训练时间的缩放方式。这为理解对比学习中的缩放行为提供了第一步理论步骤，并为平衡模型大小、数据和优化计算提供了指导。
英文摘要：
Scaling laws describe how learning performance varies with model size, data size, and compute. While recent theoretical work has established scaling laws for sketched linear regression, much less is understood for contrastive representation learning. In this paper, we study a sketched linear model for contrastive learning under a paired Gaussian latent-variable setup. The learner observes only sketched views of two correlated variables and trains a bilinear contrastive score by full-batch empirical gradient descent. We analyze a Gaussian-negative quadratic contrastive surrogate under aligned power-law spectra and a contrastive source condition, where we derive a risk decomposition into irreducible risk, approximation error, GD bias, GD variance, and a cross term. The cross term is controlled by the bias and variance and therefore does not affect the upper-bound scaling. Our main theorem gives an explicit scaling law with respect to sketch dimension $M$, sample size $N$, and effective optimization horizon $L_{\mathrm{eff}}γ$. Compared with standard linear-regression scaling laws, the contrastive setting must learn interactions between two views, and this changes how optimization and finite-sample noise scale with model size, data, and training time. This provides a first theoretical step toward understanding scaling behavior in contrastive learning and gives guidance for balancing model size, data, and optimization compute.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
