arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.07920cs.LG
收藏

InfoNCE损失可证明地学习保序表示

InfoNCE Loss Provably Learns Cluster-Preserving Representations

The University of Texas at Austin(德克萨斯大学奥斯汀分校);Google Research India(谷歌印度研究院)

Advait Parulekar, Liam Collins, Karthikeyan Shanmugam, Aryan Mokhtari, Sanjay Shakkottai

摘要:

对比学习的目标是学习一种表示,通过保持具有相似内容的样本,例如狗的“dogness”,在由表示生成的空间中彼此靠近,来保留潜在的集群。解决这个无监督学习问题的一个常见而成功的方法是最小化与训练样本相关的InfoNCE损失,其中每个样本与它们的增强(正样本,如旋转,裁剪)和一批负样本(不相关的样本)相关联。据我们所知,如果通过最小化InfoNCE损失来学习的表示保留了底层数据集群,这是没有答案的,因为它只会促进学习忠实于增强的表示,即,图像及其增强具有相同的表示。我们的主要结果是表明,在集群内的增强集可能不重叠,但相对于学习函数类的复杂性,InfoNCE用有限数量的负样本学习的表示也与数据中的集群一致。

英文摘要:

The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e.g. the ``dogness'' of a dog, close to each other in the space generated by the representation. A common and successful approach for tackling this unsupervised learning problem is minimizing the InfoNCE loss associated with the training samples, where each sample is associated with their augmentations (positive samples such as rotation, crop) and a batch of negative samples (unrelated samples). To the best of our knowledge, it was unanswered if the representation learned by minimizing the InfoNCE loss preserves the underlying data clusters, as it only promotes learning a representation that is faithful to augmentations, i.e., an image and its augmentations have the same representation. Our main result is to show that the representation learned by InfoNCE with a finite number of negative samples is also consistent with respect to clusters in the data, under the condition that the augmentation sets within clusters may be non-overlapping but are close and intertwined, relative to the complexity of the learning function class.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑