利用类内对比学习丰富知识蒸馏
Enriching Knowledge Distillation with Intra-Class Contrastive Learning
AI总结：
针对知识蒸馏中教师软标签缺乏类内多样表示的问题，本文提出带 margin loss 的类内对比学习教师训练方法，并通过理论与实验验证其有效性。
摘要：
自知识蒸馏问世以来，许多研究都聚焦于如何有效利用教师模型生成的软标签。已有研究指出，软标签中的隐式知识来源于数据中存在的多视图结构。同类样本内部的特征变化使学生模型能够通过学习多样化表示而获得更好的泛化能力。然而，在现有蒸馏方法中，教师模型主要遵循真实标签作为目标，而没有考虑同一类别内部的多样化表示。因此，我们提出在教师训练过程中引入类内对比损失，以丰富软标签所包含的类内信息。在实践中，我们发现类内损失会导致训练不稳定并减缓收敛。为缓解这些问题，我们将 margin loss（间隔损失）整合到类内对比学习中，以提高训练稳定性和收敛速度。同时，我们从理论上分析了该损失对类内距离和类间距离的影响。结果证明，类内对比损失能够丰富类内多样性。实验结果证明了所提方法的有效性。
英文摘要：
Since the advent of knowledge distillation, much research has focused on how the soft labels generated by the teacher model can be utilized effectively. Existing studies points out that the implicit knowledge within soft labels originates from the multi-view structure present in the data. Feature variations within samples of the same class allow the student model to generalize better by learning diverse representations. However, in existing distillation methods, teacher models predominantly adhere to ground-truth labels as targets, without considering the diverse representations within the same class. Therefore, we propose incorporating an intra-class contrastive loss during teacher training to enrich the intra-class information contained in soft labels. In practice, we find that intra-class loss causes instability in training and slows convergence. To mitigate these issues, margin loss is integrated into intra-class contrastive learning to improve the training stability and convergence speed. Simultaneously, we theoretically analyze the impact of this loss on the intra-class distances and inter-class distances. It has been proved that the intra-class contrastive loss can enrich the intra-class diversity. Experimental results demonstrate the effectiveness of the proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
