arXiv 2507.111612025-07-17stat.MLcs.LG

标注误差如何影响对比学习？一个来自数据降维的视角

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction

College of Informatics, Huazhong Agricultural University, Wuhan, China（华中农业大学信息学院）；Engineering Research Center of Intelligent Technology for Agriculture, Ministry of Education, China（农业智能技术工程研究中心）；College of Engineering, Huazhong Agricultural University, Wuhan, China（华中农业大学工程学院）；School of Mathematics and Statistics, University of Sydney, NSW, Australia（悉尼大学数学与统计学学院）

Jun Chen, Hong Chen, Yonghua Yu, Yiming Ying

备注：: Published as ICML2025 poster. The arXiv version is a modified version

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文从数据降维视角理论分析了标注误差对对比学习下游分类性能的负面影响，并提出结合SVD、适中嵌入维度、数据膨胀与弱增强来减少假阳性样本、保持图连通性，从而提升模型性能。

摘要：

近年来，对比学习在自监督表示学习领域取得了最先进的性能。许多先前的工作试图为对比学习的成功提供理论解释。几乎所有这些工作都依赖于一个默认假设，即标签一致性假设，但由于常见增强策略（如随机缩放裁剪 RRC）的强度和随机性，该假设在实践中可能并不成立（失败的概率被称为标注误差）。本文研究了标注误差对对比学习下游分类性能的理论影响。我们首先揭示了标注误差对下游分类风险的若干显著负面影响。为了减轻这些影响，我们将数据降维方法（例如奇异值分解 SVD）应用于原始数据以减少假阳性样本，并建立了理论和实证评估。此外，我们还发现 SVD 是一把双刃剑，由于增强图的连通性降低，它可能导致下游分类精度的恶化。基于上述观察，我们给出增强建议：应使用适中的嵌入维度（例如我们实验中的 512、1024）、数据膨胀、弱增强和 SVD，以确保较大的图连通性和较小的标注误差，从而提高模型性能。

英文摘要：

In recent years, contrastive learning has achieved state-of-the-art performance in the territory of self-supervised representation learning. Many previous works have attempted to provide the theoretical understanding underlying the success of contrastive learning. Almost all of them rely on a default assumption, i.e., the label consistency assumption, which may not hold in practice (the probability of failure is called labeling error) due to the strength and randomness of common augmentation strategies, such as random resized crop (RRC). This paper investigates the theoretical impact of labeling error on the downstream classification performance of contrastive learning. We first reveal several significant negative impacts of labeling error on downstream classification risk. To mitigate these impacts, data dimensionality reduction method (e.g., singular value decomposition, SVD) is applied on original data to reduce false positive samples, and establish both theoretical and empirical evaluations. Moreover, it is also found that SVD acts as a double-edged sword, which may lead to the deterioration of downstream classification accuracy due to the reduced connectivity of the augmentation graph. Based on the above observations, we give the augmentation suggestion that we should use some moderate embedding dimension (such as $512, 1024$ in our experiments), data inflation, weak augmentation, and SVD to ensure large graph connectivity and small labeling error to improve model performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。