WERank：基于权重正则化防止自监督学习秩退化研究
WERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization
AI总结：
针对自监督学习中的维度坍塌问题，提出权重正则化器WERank，通过约束网络不同层权重防止秩退化，在图SSL等任务中有效提升BYOL的表征秩与下游准确率。
摘要：
限制自监督学习（SSL）表征质量的一个常见现象是维度坍塌（也称秩退化），即学习到的表征被映射到表征空间的低维子空间中。最先进的SSL方法已被证明会遭受维度坍塌，无法维持满秩。为防止该问题，近期方法提出使用对比损失、正则化技术或架构技巧。我们提出了WERank，一种作用于网络权重参数的新型正则化器，用于防止网络不同层发生秩退化。我们提供了经验证据和数学证明，以展示所提正则化方法在防止维度坍塌方面的有效性。我们在图SSL上验证了WERank的影响，由于缺乏适当的数据增强，维度坍塌在该任务中更为显著。我们经验性地证明，WERank能有效帮助BYOL在SSL预训练期间达到更高的秩，从而在评估探测中提升下游准确率。消融研究和实验分析揭示了所提方法性能提升背后的潜在因素。
英文摘要：
A common phenomena confining the representation quality in Self-Supervised Learning (SSL) is dimensional collapse (also known as rank degeneration), where the learned representations are mapped to a low dimensional subspace of the representation space. The State-of-the-Art SSL methods have shown to suffer from dimensional collapse and fall behind maintaining full rank. Recent approaches to prevent this problem have proposed using contrastive losses, regularization techniques, or architectural tricks. We propose WERank, a new regularizer on the weight parameters of the network to prevent rank degeneration at different layers of the network. We provide empirical evidence and mathematical justification to demonstrate the effectiveness of the proposed regularization method in preventing dimensional collapse. We verify the impact of WERank on graph SSL where dimensional collapse is more pronounced due to the lack of proper data augmentation. We empirically demonstrate that WERank is effective in helping BYOL to achieve higher rank during SSL pre-training and consequently downstream accuracy during evaluation probing. Ablation studies and experimental analysis shed lights on the underlying factors behind the performance gains of the proposed approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
