DisCo：轻量级模型上的改进自监督学习与蒸馏对比学习
DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
摘要：
虽然自监督表示学习（SSL）已经受到了社会的广泛关注，最近的研究认为，其性能将遭受悬崖下降时，模型大小减小。目前的方法主要依靠对比学习来训练网络，在这项工作中，我们提出了一个简单而有效的蒸馏对比学习（DisCo），以缓解问题的大幅度。具体来说，我们发现通过主流SSL方法获得的最终嵌入包含最丰富的信息，并建议通过约束学生的最后嵌入与教师的最后嵌入来提取最终嵌入，以最大限度地将教师的知识传输到轻量级模型。此外，在实验中，我们发现存在一种称为蒸馏瓶颈的现象，并提出了扩大嵌入维数来缓解这一问题。我们的方法在部署过程中没有向轻量级模型引入任何额外的参数。实验结果表明，我们的方法达到了国家的最先进的轻量级模型。特别地，当使用ResNet-101/ResNet-50作为教师来教导EfficientNet-B 0时，EfficientNet-B 0在ImageNet上的线性结果非常接近ResNet-101/ResNet-50，但EfficientNet-B 0的参数数量仅为ResNet-101/ResNet-50的9.4\%/16.3\%。代码可以在https：//github上找到。com/Yuting-Gao/DisCo-pytorch.
英文摘要：
While self-supervised representation learning (SSL) has received widespread attention from the community, recent research argue that its performance will suffer a cliff fall when the model size decreases. The current method mainly relies on contrastive learning to train the network and in this work, we propose a simple yet effective Distilled Contrastive Learning (DisCo) to ease the issue by a large margin. Specifically, we find the final embedding obtained by the mainstream SSL methods contains the most fruitful information, and propose to distill the final embedding to maximally transmit a teacher's knowledge to a lightweight model by constraining the last embedding of the student to be consistent with that of the teacher. In addition, in the experiment, we find that there exists a phenomenon termed Distilling BottleNeck and present to enlarge the embedding dimension to alleviate this problem. Our method does not introduce any extra parameter to lightweight models during deployment. Experimental results demonstrate that our method achieves the state-of-the-art on all lightweight models. Particularly, when ResNet-101/ResNet-50 is used as teacher to teach EfficientNet-B0, the linear result of EfficientNet-B0 on ImageNet is very close to ResNet-101/ResNet-50, but the number of parameters of EfficientNet-B0 is only 9.4\%/16.3\% of ResNet-101/ResNet-50. Code is available at https://github. com/Yuting-Gao/DisCo-pytorch.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
