非线性的边缘：什么使BYOL滴答声的简单视图
The Edge of Orthogonality: A Simple View of What Makes BYOL Tick
摘要：
自我预测的无监督学习方法，如BYOL或SimSiam，已经显示出令人印象深刻的结果，并且与直觉相反，不会崩溃到琐碎的表示。在这项工作中，我们的目标是探索最简单的可能的数学论据，以解释自我预测无监督学习背后的潜在机制。我们首先观察到，这些方法主要依赖于预测网络（和停止梯度）的存在。通过简单的线性代数，我们证明了当使用线性预测器时，最佳预测器接近正交投影，并提出了一个基于正交归一化的一般框架，该框架能够解释并直观地解释BYOL工作的原因。此外，该框架还证明了指数移动平均和停止梯度算子在BYOL中作为一种有效的正交归一化机制的关键作用。我们使用这些见解，提出四个新的\emph{closed-form predictor}变体BYOL，以支持我们的分析。我们的封闭形式预测器在$100$和$300$时期（ImageNet上的顶级$1$线性准确度）优于标准线性可训练预测器BYOL。
英文摘要：
Self-predictive unsupervised learning methods such as BYOL or SimSiam have shown impressive results, and counter-intuitively, do not collapse to trivial representations. In this work, we aim at exploring the simplest possible mathematical arguments towards explaining the underlying mechanisms behind self-predictive unsupervised learning. We start with the observation that those methods crucially rely on the presence of a predictor network (and stop-gradient). With simple linear algebra, we show that when using a linear predictor, the optimal predictor is close to an orthogonal projection, and propose a general framework based on orthonormalization that enables to interpret and give intuition on why BYOL works. In addition, this framework demonstrates the crucial role of the exponential moving average and stop-gradient operator in BYOL as an efficient orthonormalization mechanism. We use these insights to propose four new \emph{closed-form predictor} variants of BYOL to support our analysis. Our closed-form predictors outperform standard linear trainable predictor BYOL at $100$ and $300$ epochs (top-$1$ linear accuracy on ImageNet).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
