BicKD：双侧对比知识蒸馏
BicKD: Bilateral Contrastive Knowledge Distillation
AI总结：
本文提出BicKD，通过双侧对比损失增强知识迁移，改进传统知识蒸馏的样本级和类别级对比能力，提升预测分布几何结构的正则化效果。
摘要：
知识蒸馏（KD）是一种将知识从教师模型转移到学生模型的机器学习框架。Hinton等人提出的 vanilla KD 是 logit 基础蒸馏的主流方法，表现优异，但仅在样本级对教师和学生预测进行概率对齐，缺乏类别级比较机制。此外，vanilla KD 对概率空间无结构约束。本文提出一种简单有效的双侧对比知识蒸馏（BicKD）方法。该方法引入新颖的双侧对比损失，强化不同类别泛化空间之间的正交性，同时保持同一类别内的一致性。双侧公式使教师和学生在样本级和类别级预测模式之间进行显式比较。通过强调概率正交性，BicKD 进一步正则化预测分布的几何结构。大量实验表明，BicKD 方法增强了知识迁移效果，并在各种模型架构和基准上一致优于最先进的知识蒸馏技术。
英文摘要：
Knowledge distillation (KD) is a machine learning framework that transfers knowledge from a teacher model to a student model. The vanilla KD proposed by Hinton et al. has been the dominant approach in logit-based distillation and demonstrates compelling performance. However, it only performs sample-wise probability alignment between teacher and student's predictions, lacking an mechanism for class-wise comparison. Besides, vanilla KD imposes no structural constraint on the probability space. In this work, we propose a simple yet effective methodology, bilateral contrastive knowledge distillation (BicKD). This approach introduces a novel bilateral contrastive loss, which intensifies the orthogonality among different class generalization spaces while preserving consistency within the same class. The bilateral formulation enables explicit comparison of both sample-wise and class-wise prediction patterns between teacher and student. By emphasizing probabilistic orthogonality, BicKD further regularizes the geometric structure of the predictive distribution. Extensive experiments show that our BicKD method enhances knowledge transfer, and consistently outperforms state-of-the-art knowledge distillation techniques across various model architectures and benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
