通过跨模态教师融合丰富知识蒸馏
Enriching Knowledge Distillation with Cross-Modal Teacher Fusion
AI总结：
针对现有多教师知识蒸馏缺乏知识多样性的问题，提出RichKD框架，融合传统教师模型与CLIP的跨模态知识，结合多提示文本引导，提升预测可靠性与类间一致性，在多基准及鲁棒性测试中优于基线。
摘要：
多教师知识蒸馏（KD）是一种比传统单教师方法更有效的技术，它通过logit或特征匹配将知识从专家教师模型迁移到紧凑的学生模型。然而，大多数现有方法缺乏知识多样性，因为它们仅依赖单模态视觉信息，忽略了跨模态表示的潜力。在这项工作中，我们探索将CLIP的视觉-语言知识作为KD的补充监督源，这一领域在很大程度上仍未被充分研究。我们提出了一个简单但有效的框架，将传统教师模型的logits和特征与CLIP的logits和特征进行融合。通过结合CLIP的多提示文本引导，融合后的监督信息既捕捉了数据集特定的视觉线索，又包含了语义丰富的视觉线索。除了精度之外，分析表明融合后的教师模型能产生更自信、更可靠的预测，显著增加了自信正确的案例，同时减少了自信错误的案例。此外，与CLIP的融合优化了整个logit分布，为非目标类别生成了语义上有意义的概率，从而提高了类间一致性和蒸馏质量。尽管方法简单，所提出的Enriching Knowledge Distillation（RichKD）方法在多个基准测试中持续优于大多数现有基线，并在分布偏移和输入损坏情况下表现出更强的鲁棒性。
英文摘要：
Multi-teacher knowledge distillation (KD), a more effective technique than traditional single-teacher methods, transfers knowledge from expert teachers to a compact student model using logit or feature matching. However, most existing approaches lack knowledge diversity, as they rely solely on unimodal visual information, overlooking the potential of cross-modal representations. In this work, we explore the use of CLIP's vision-language knowledge as a complementary source of supervision for KD, an area that remains largely underexplored. We propose a simple yet effective framework that fuses the logits and features of a conventional teacher with those from CLIP. By incorporating CLIP's multi-prompt textual guidance, the fused supervision captures both dataset-specific and semantically enriched visual cues. Beyond accuracy, analysis shows that the fused teacher yields more confident and reliable predictions, significantly increasing confident-correct cases while reducing confidently wrong ones. Moreover, fusion with CLIP refines the entire logit distribution, producing semantically meaningful probabilities for non-target classes, thereby improving inter-class consistency and distillation quality. Despite its simplicity, the proposed method, Enriching Knowledge Distillation (RichKD), consistently outperforms most existing baselines across multiple benchmarks and exhibits stronger robustness under distribution shifts and input corruptions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
