arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.09252cs.LG
收藏

关于嵌入范数在自监督学习中的重要性

On the Importance of Embedding Norms in Self-Supervised Learning

University of Amsterdam(阿姆斯特丹大学)

Andrew Draganov, Sharvaree Vadgama, Sebastian Damrich, Jan Niklas Böhm, Lucas Maes, Dmitry Kobak, Erik Bekkers

AI总结:

本文研究了嵌入范数在自监督学习中的作用,通过理论分析和实验表明范数影响收敛速度和网络置信度,且较小的范数对应意外样本。

摘要:

自监督学习(SSL)允许在无监督信号的情况下训练数据表示,已成为机器学习的重要范式。大多数SSL方法使用嵌入向量的余弦相似度,从而有效将数据嵌入到超球面上。虽然这似乎表明嵌入范数在SSL中不起作用,但一些近期工作表明嵌入范数与网络收敛和置信度有关。本文解决这一明显矛盾,系统地确立嵌入范数在SSL训练中的作用。通过理论分析、模拟和实验,我们证明嵌入范数(i)控制SSL收敛速度(ii)编码网络置信度,较小的范数对应意外样本。此外,我们还表明操纵嵌入范数对收敛速度有显著影响。我们的发现表明,SSL嵌入范数对于理解和优化网络行为至关重要。

英文摘要:

Self-supervised learning (SSL) allows training data representations without a supervised signal and has become an important paradigm in machine learning. Most SSL methods employ the cosine similarity between embedding vectors and hence effectively embed data on a hypersphere. While this seemingly implies that embedding norms cannot play any role in SSL, a few recent works have suggested that embedding norms have properties related to network convergence and confidence. In this paper, we resolve this apparent contradiction and systematically establish the embedding norm's role in SSL training. Using theoretical analysis, simulations, and experiments, we show that embedding norms (i) govern SSL convergence rates and (ii) encode network confidence, with smaller norms corresponding to unexpected samples. Additionally, we show that manipulating embedding norms can have large effects on convergence speed. Our findings demonstrate that SSL embedding norms are integral to understanding and optimizing network behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑