带引导式停止梯度的隐式对比表示学习
Implicit Contrastive Representation Learning with Guided Stop-gradient
AI总结：
本文提出引导式停止梯度方法，在SimSiam和BYOL等非对称正样本自监督框架中隐式实现对比学习，以稳定训练、提升性能并避免小批量或无预测器时的坍缩。
摘要：
在自监督表示学习中，孪生网络是一种通过拉近正样本对的表示来学习变换不变性的自然架构，但它容易坍缩为退化解。为解决该问题，对比学习使用对比损失，通过推开负样本对的表示来防止坍缩。但已知采用负采样的算法对负样本数量减少并不鲁棒。因此，另一方面，存在不使用负样本对的算法。许多仅使用正样本的算法采用由源编码器和目标编码器组成的非对称网络架构，作为应对坍缩的关键因素。通过利用这种非对称架构，我们引入了一种隐式融入对比学习思想的方法论。作为其实现，我们提出了一种新方法——引导式停止梯度（guided stop-gradient）。我们将该方法应用于基准算法 SimSiam 和 BYOL，并表明我们的方法能稳定训练并提升性能。我们还表明，采用该方法的算法在小批量大小下表现良好，即使没有预测器也不会坍缩。代码可在 https://github.com/bych-lee/gsg 获取。
英文摘要：
In self-supervised representation learning, Siamese networks are a natural architecture for learning transformation-invariance by bringing representations of positive pairs closer together. But it is prone to collapse into a degenerate solution. To address the issue, in contrastive learning, a contrastive loss is used to prevent collapse by moving representations of negative pairs away from each other. But it is known that algorithms with negative sampling are not robust to a reduction in the number of negative samples. So, on the other hand, there are algorithms that do not use negative pairs. Many positive-only algorithms adopt asymmetric network architecture consisting of source and target encoders as a key factor in coping with collapse. By exploiting the asymmetric architecture, we introduce a methodology to implicitly incorporate the idea of contrastive learning. As its implementation, we present a novel method guided stop-gradient. We apply our method to benchmark algorithms SimSiam and BYOL and show that our method stabilizes training and boosts performance. We also show that the algorithms with our method work well with small batch sizes and do not collapse even when there is no predictor. The code is available at https://github.com/bych-lee/gsg.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
