特征归一化可防止非对比学习动力学的坍缩
Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics
AI总结：
该研究针对非对比学习的表示坍缩问题，通过引入含特征归一化的余弦损失拓展了现有动力学理论，发现余弦损失诱导的六阶动力学生成稳定平衡点，证明特征归一化可鲁棒防止动力学坍缩。
摘要：
对比学习是一种自监督表示学习框架，该框架通过数据增强生成的两个正样本视图在数据表示空间中受吸引力作用而变得相似，同时受排斥力作用而与负样本远离。以BYOL和SimSiam为代表的非对比学习进一步去除了负样本，提升了计算效率。乍看之下，由于缺乏排斥力，学习到的表示可能会坍缩到单个点，但Tian等人（2021）通过学习动力学分析揭示，若数据增强的强度远高于正则化强度，表示即可避免坍缩。然而，他们的分析未考虑常用的特征归一化——这是衡量表示相似性前的归一化步骤，因此过强的正则化可能导致动力学坍缩，这在存在特征归一化的情况下是一种不符合实际的行为。为此，我们通过引入包含特征归一化的余弦损失，拓展了此前基于L2损失的理论。我们证明，余弦损失会诱导六阶动力学（而L2损失诱导三阶动力学），在该动力学中，即使给定初始参数下仅存在坍缩解，也会动态出现一个稳定平衡点。由此我们提出了一种新的认知：特征归一化在鲁棒防止动力学坍缩方面发挥着重要作用。
英文摘要：
Contrastive learning is a self-supervised representation learning framework, where two positive views generated through data augmentation are made similar by an attraction force in a data representation space, while a repulsive force makes them far from negative examples. Non-contrastive learning, represented by BYOL and SimSiam, further gets rid of negative examples and improves computational efficiency. While learned representations may collapse into a single point due to the lack of the repulsive force at first sight, Tian et al. (2021) revealed through the learning dynamics analysis that the representations can avoid collapse if data augmentation is sufficiently stronger than regularization. However, their analysis does not take into account commonly-used feature normalization, a normalizer before measuring the similarity of representations, and hence excessively strong regularization may collapse the dynamics, which is an unnatural behavior under the presence of feature normalization. Therefore, we extend the previous theory based on the L2 loss by considering the cosine loss, which involves feature normalization. We show that the cosine loss induces sixth-order dynamics (while the L2 loss induces a third-order one), in which a stable equilibrium dynamically emerges even if there are only collapsed solutions with given initial parameters. Thus, we offer a new understanding that feature normalization plays an important role in robustly preventing the dynamics collapse.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
