arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.03338cs.LGcs.CV
收藏

IdEst: 通过内在维度评估自监督学习表示

IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimension

University of California, Berkeley(加州大学伯克利分校)

Julie Mordacq, Vicky Kalogeiton, Steve Oudot

AI总结:

提出IdEst方法,利用最小生成树维度估计器评估自监督学习表示的内在维度,发现其与下游线性探测性能强相关,并能高效选择超参数。

摘要:

自监督学习(SSL)已成为从无标签数据中学习有意义表示的有效范式。然而,评估这些表示的标准协议——线性探测——计算成本高、对超参数敏感,并且对表示空间的几何结构提供的洞察有限。在这项工作中,受神经网络泛化与内在维度(ID)之间联系的启发,我们提出了IdEst,一种通过最小生成树维度估计器($\mathrm{dim}_\mathrm{MST}$)估计SSL表示ID的方法。在多种数据集、架构和SSL预训练目标上,我们表明IdEst与下游线性探测性能强相关。此外,我们证明IdEst能够实现高效的超参数选择,与监督替代方案相比显著降低计算成本。我们的结果突出了内在维度作为评估SSL表示的原则性几何代理,补充了标准的监督探测协议。

英文摘要:

Self-supervised learning (SSL) has emerged as a powerful paradigm for learning meaningful representations from unlabeled data. However, the standard protocol for evaluating these representations, linear probing, is computationally expensive, sensitive to hyperparameters, and provides limited insight into the geometric structure of the representation space. In this work, motivated by connections between neural network generalization and intrinsic dimension (ID) we propose IdEst, a method for estimating the ID of SSL representations via the Minimum Spanning Tree dimension estimator ($\mathrm{dim}_\mathrm{MST}$). Across diverse datasets, architectures, and SSL pretraining objectives, we show that IdEst strongly correlates with downstream linear probe performances. Furthermore, we demonstrate that IdEst enables efficient hyperparameter selection, significantly reducing the computational cost compared to supervised alternatives. Our results highlight intrinsic dimensionality as a principled geometric proxy for assessing SSL representations, complementing standard supervised probing protocols.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑