揭示对比学习中的投影头：来自扩张与收缩的洞见
Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkage
AI总结：
本文从理论与实验上分析对比学习中线性投影器的扩张和收缩效应，并用线性变换刻画其对下游线性分类准确率的影响。
摘要：
我们研究了对比学习所使用的 encoder-projector（编码器-投影器）框架（例如 SimCLR）中投影头（也称为 projector）的作用。我们的目标是解释一个已观察到的现象：以线性分类的下游准确率衡量，投影器之前学到的表示优于投影器之后学到的表示，即使投影器本身是线性的也是如此。在本文中，我们为实现这一目标作出两项重要贡献。首先，通过实证和理论分析，我们识别出对比损失在投影器上诱发的两个关键效应——扩张与收缩。本质上，对比损失会扩张或收缩编码器所学表示中的信号方向，具体取决于增强强度、对比损失中使用的温度等因素。其次，受扩张与收缩现象启发，我们提出一族线性变换，用于精确建模投影器的行为。这使我们能够在高维渐近极限下精确刻画下游线性分类准确率。我们的发现表明，在收缩（或扩张）区间运行的线性投影器会阻碍（或提升）下游分类准确率。这首次为（线性）投影器为何会影响所学表示的下游性能提供了理论解释。我们在合成数据和真实图像数据上进行的大量实验进一步证实了这些理论发现。
英文摘要：
We investigate the role of projection heads, also known as projectors, within the encoder-projector framework (e.g., SimCLR) used in contrastive learning. We aim to demystify the observed phenomenon where representations learned before projectors outperform those learned after -- measured using the downstream linear classification accuracy, even when the projectors themselves are linear. In this paper, we make two significant contributions towards this aim. Firstly, through empirical and theoretical analysis, we identify two crucial effects -- expansion and shrinkage -- induced by the contrastive loss on the projectors. In essence, contrastive loss either expands or shrinks the signal direction in the representations learned by an encoder, depending on factors such as the augmentation strength, the temperature used in contrastive loss, etc. Secondly, drawing inspiration from the expansion and shrinkage phenomenon, we propose a family of linear transformations to accurately model the projector's behavior. This enables us to precisely characterize the downstream linear classification accuracy in the high-dimensional asymptotic limit. Our findings reveal that linear projectors operating in the shrinkage (or expansion) regime hinder (or improve) the downstream classification accuracy. This provides the first theoretical explanation as to why (linear) projectors impact the downstream performance of learned representations. Our theoretical findings are further corroborated by extensive experiments on both synthetic data and real image data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
