渐进式类级蒸馏
Progressive Class-level Distillation
AI总结：
针对logit蒸馏中高置信度类别主导导致知识转移不充分的问题，提出Progressive Class-level Distillation（PCD）方法，通过基于师生logits差异排序的分阶段双向蒸馏，实现由细到粗和由粗到细的渐进式知识转移，在分类和检测任务上优于现有方法。
摘要：
在知识蒸馏（KD）中，logit蒸馏（LD）旨在通过在logits级别进行准确的师生对齐，将类级知识从更强大的教师网络转移到小型学生模型。由于高置信度的目标类别通常主导蒸馏过程，在传统方法中，同样包含判别信息的低概率类别被忽视，导致知识转移不充分。为了解决这个问题，我们提出了一种简单而有效的LD方法，称为Progressive Class-level Distillation（PCD）。与执行全类别集成蒸馏的现有方法不同，我们的PCD方法执行分阶段蒸馏以逐步进行知识转移。更具体地说，我们对师生logits差异进行排序以从头识别蒸馏优先级，随后将整个LD过程划分为多个阶段。接下来，进行双向分阶段蒸馏，结合由细到粗的渐进学习和反向由粗到细的细化，通过在不同蒸馏阶段的独立类别组内进行充分的logits对齐，实现全面的知识转移。在公共基准数据集上的扩展实验证明了我们的方法在分类和检测任务中优于现有技术。
英文摘要：
In knowledge distillation (KD), logit distillation (LD) aims to transfer class-level knowledge from a more powerful teacher network to a small student model via accurate teacher-student alignment at the logits level. Since high-confidence object classes usually dominate the distillation process, low-probability classes which also contain discriminating information are downplayed in conventional methods, leading to insufficient knowledge transfer. To address this issue, we propose a simple yet effective LD method termed Progressive Class-level Distillation (PCD). In contrast to existing methods which perform all-class ensemble distillation, our PCD approach performs stage-wise distillation for step-by-step knowledge transfer. More specifically, we perform ranking on teacher-student logits difference for identifying distillation priority from scratch, and subsequently divide the entire LD process into multiple stages. Next, bidirectional stage-wise distillation incorporating fine-to-coarse progressive learning and reverse coarse-to-fine refinement is conducted, allowing comprehensive knowledge transfer via sufficient logits alignment within separate class groups in different distillation stages. Extension experiments on public benchmarking datasets demonstrate the superiority of our method compared to state-of-the-arts for both classification and detection tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
