自监督模型的表示学习动力学
Representation Learning Dynamics of Self-Supervised Models
AI总结：
本文研究自监督表示学习动力学，发现朴素扩展导致维度坍缩，提出正交约束并推导格拉斯曼流形上梯度下降的精确动力学，为SSL理论分析提供框架。
摘要：
自监督学习（SSL）是从未标记数据中学习表示的重要范式，基于神经网络的SSL在实践中已取得巨大成功。然而，目前对SSL的理论分析大多局限于泛化误差界。相比之下，学习动力学通常能精确刻画基于神经网络的模型行为，但迄今为止主要局限于监督设置。本文研究了SSL模型的学习动力学，特别是通过最小化对比损失和非对比损失获得的表示。我们表明，将多元回归的动力学朴素扩展到SSL会导致学习平凡的标量表示，从而在SSL中表现出维度坍缩。因此，我们提出了对权重施加正交性约束的SSL目标，并推导了在格拉斯曼流形上使用梯度下降训练的SSL模型的精确（与网络宽度无关的）学习动力学。我们还论证了SSL模型的无限宽度近似显著偏离监督模型的神经正切核近似。我们通过数值实验说明了理论结果的有效性，并讨论了所提出的结果如何为对比和非对比SSL的进一步理论分析提供框架。
英文摘要：
Self-Supervised Learning (SSL) is an important paradigm for learning representations from unlabelled data, and SSL with neural networks has been highly successful in practice. However current theoretical analysis of SSL is mostly restricted to generalisation error bounds. In contrast, learning dynamics often provide a precise characterisation of the behaviour of neural networks based models but, so far, are mainly known in supervised settings. In this paper, we study the learning dynamics of SSL models, specifically representations obtained by minimising contrastive and non-contrastive losses. We show that a naive extension of the dymanics of multivariate regression to SSL leads to learning trivial scalar representations that demonstrates dimension collapse in SSL. Consequently, we formulate SSL objectives with orthogonality constraints on the weights, and derive the exact (network width independent) learning dynamics of the SSL models trained using gradient descent on the Grassmannian manifold. We also argue that the infinite width approximation of SSL models significantly deviate from the neural tangent kernel approximations of supervised models. We numerically illustrate the validity of our theoretical findings, and discuss how the presented results provide a framework for further theoretical analysis of contrastive and non-contrastive SSL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
