通过相似性结构理解对比学习机制：一项理论分析
Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis
AI总结：
本文提出 KCL 框架并以统计依赖刻画表征相似性结构，证明其能描述对比学习形成的聚类，同时给出下游分类误差、泛化误差及代理泛化保证。
摘要：
对比学习是自监督表征学习的一种高效方法。尽管近期研究在对比学习的理论理解方面取得了进展，但关于如何刻画所学表征聚类的研究仍然有限。本文旨在从理论角度阐明这一刻画问题。为此，作者考虑了一个基于核的对比学习框架，称为 Kernel Contrastive Learning（KCL，核对比学习）；在将理论结果应用于其他框架时，核函数发挥着重要作用。作者利用统计依赖的观点，提出了所学表征相似性结构的一种表述形式，并通过该表述研究了基于核的对比损失的理论性质。作者首先证明，该表述能够刻画由基于核的对比学习框架所学到的表征结构。随后给出了下游任务分类误差的一个新上界，解释了其理论与对比学习在经验上的成功相一致。作者还建立了 KCL 的泛化误差界。最后，通过一个代理界，给出了 KCL 对下游分类任务泛化能力的保证。
英文摘要：
Contrastive learning is an efficient approach to self-supervised representation learning. Although recent studies have made progress in the theoretical understanding of contrastive learning, the investigation of how to characterize the clusters of the learned representations is still limited. In this paper, we aim to elucidate the characterization from theoretical perspectives. To this end, we consider a kernel-based contrastive learning framework termed Kernel Contrastive Learning (KCL), where kernel functions play an important role when applying our theoretical results to other frameworks. We introduce a formulation of the similarity structure of learned representations by utilizing a statistical dependency viewpoint. We investigate the theoretical properties of the kernel-based contrastive loss via this formulation. We first prove that the formulation characterizes the structure of representations learned with the kernel-based contrastive learning framework. We show a new upper bound of the classification error of a downstream task, which explains that our theory is consistent with the empirical success of contrastive learning. We also establish a generalization error bound of KCL. Finally, we show a guarantee for the generalization ability of KCL to the downstream classification task via a surrogate bound.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
