对比学习学到了哪些特征？论简单性偏差在类别坍缩与特征抑制中的作用
Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression
AI总结：
本文首次提出统一理论框架，揭示梯度下降的简单性偏差导致对比学习中的类别坍缩和特征抑制，并给出增加嵌入维度与改进数据增强两种解决方案，同时解释了监督与无监督对比学习结合能提升表示质量的原因。
摘要：
对比学习（CL）已成为一种强大的表示学习技术，无论有无标签监督。然而，有监督的对比学习容易因未能捕获类内子类的所有特征而导致子类表示坍缩，而无监督对比学习可能因专注于学习容易的类无关特征而抑制较难的类相关特征；两者都显著损害表示质量。然而，目前对测试阶段的类别坍缩或特征抑制尚无理论理解。我们提供了第一个统一且理论上严格的框架，以确定对比学习学到了哪些特征。我们的分析表明，或许令人惊讶的是，（随机）梯度下降倾向于寻找更简单解的偏差是导致子类表示坍缩和抑制较难类相关特征的关键因素。此外，我们提出增加嵌入维度和提高数据增强质量作为两个有理论依据的解决特征抑制的方案。我们还首次从理论上解释了为什么同时使用有监督和无监督对比学习能获得更高质量的表示，即使在使用常用的随机梯度方法时也是如此。
英文摘要：
Contrastive learning (CL) has emerged as a powerful technique for representation learning, with or without label supervision. However, supervised CL is prone to collapsing representations of subclasses within a class by not capturing all their features, and unsupervised CL may suppress harder class-relevant features by focusing on learning easy class-irrelevant features; both significantly compromise representation quality. Yet, there is no theoretical understanding of \textit{class collapse} or \textit{feature suppression} at \textit{test} time. We provide the first unified theoretically rigorous framework to determine \textit{which} features are learnt by CL. Our analysis indicate that, perhaps surprisingly, bias of (stochastic) gradient descent towards finding simpler solutions is a key factor in collapsing subclass representations and suppressing harder class-relevant features. Moreover, we present increasing embedding dimensionality and improving the quality of data augmentations as two theoretically motivated solutions to {feature suppression}. We also provide the first theoretical explanation for why employing supervised and unsupervised CL together yields higher-quality representations, even when using commonly-used stochastic gradient methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
