非对比自监督学习的双重视角
Dual Perspectives on Non-Contrastive Self-Supervised Learning
AI总结：
本文从优化和动力系统双重视角分析非对比自监督学习中停止梯度和指数移动平均的机制，揭示其避免表征崩溃的理论依据，并通过实验验证其稳定性。
摘要：
停止梯度和指数移动平均迭代过程常用于非对比方法中自监督学习，以避免表征崩溃，在实际下游应用中表现出色。本报告从优化和动力系统两个视角调查这些过程。我们证明，尽管它们不优化原始目标或任何其他光滑函数，但它们确实避免了崩溃。跟随Tian等人的研究，但不使用他们在证明中所用的额外假设，我们从动力系统视角证明，在线性情况下，不使用停止梯度或指数移动平均最小化原始目标函数会导致崩溃。相反，我们明确表征了这些两种过程所关联的动力系统的平衡点，并在参数空间中作为代数簇进行描述，证明它们通常渐近稳定。我们的理论发现通过实际和合成数据的实验证明。
英文摘要：
The {\em stop gradient} and {\em exponential moving average} iterative procedures are commonly used in non-contrastive approaches to self-supervised learning to avoid representation collapse, with excellent performance in downstream applications in practice. This presentation investigates these procedures from the dual viewpoints of optimization and dynamical systems. We show that, in general, although they {\em do not} optimize the original objective, or {\em any} other smooth function, they {\em do} avoid collapse Following~\citet{Tian21}, but without any of the extra assumptions used in their proofs, we then show using a dynamical system perspective that, in the linear case, minimizing the original objective function without the use of a stop gradient or exponential moving average {\em always} leads to collapse. Conversely, we characterize explicitly the equilibria of the dynamical systems associated with these two procedures in this linear setting as algebraic varieties in their parameter space, and show that they are, in general, {\em asymptotically stable}. Our theoretical findings are illustrated by empirical experiments with real and synthetic data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
