TopKD：顶缩放知识蒸馏
TopKD: Top-scaled Knowledge Distillation
AI总结：
提出TopKD框架，通过Top-K缩放模块和解耦损失增强基于logit的蒸馏，在多个数据集上超越现有方法，并适用于视觉Transformer。
摘要：
知识蒸馏（KD）的最新进展主要强调特征层面的知识迁移，常常忽视了教师模型logit分布中嵌入的关键信息。在本文中，我们重新审视基于logit的蒸馏，并揭示了一个未被充分探索但至关重要的元素：Top-K知识。受此启发，我们提出了顶缩放知识蒸馏（TopKD），这是一个简单、高效且与架构无关的框架，显著增强了基于logit的蒸馏。TopKD由两个主要组件组成：（1）Top-K缩放模块（TSM），自适应地放大信息量最大的logit；（2）Top-K解耦损失（TDL），提供有针对性的有效监督。值得注意的是，TopKD无需引入额外模块或要求架构更改，即可无缝集成到现有KD方法中。在CIFAR-100、ImageNet、STL-10和Tiny-ImageNet上的大量实验表明，TopKD持续超越最先进的蒸馏方法。此外，我们的方法在蒸馏视觉Transformer时表现出显著的有效性，突显了其在不同网络架构中的通用性。这些发现凸显了logit在推进知识蒸馏方面的巨大潜力。
英文摘要：
Recent advances in knowledge distillation (KD) predominantly emphasize feature-level knowledge transfer, frequently overlooking critical information embedded within the teacher's logit distributions. In this paper, we revisit logit-based distillation and reveal an underexplored yet critical element: Top-K knowledge. Motivated by this insight, we propose Top-scaled Knowledge Distillation (TopKD), a simple, efficient, and architecture-agnostic framework that significantly enhances logit-based distillation. TopKD consists of two main components: (1) a Top-K Scaling Module (TSM), which adaptively amplifies the most informative logits, and (2) a Top-K Decoupled Loss (TDL), which offers targeted and effective supervision. Notably, TopKD integrates seamlessly into existing KD methods without introducing extra modules or requiring architectural changes. Extensive experiments on CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet demonstrate that TopKD consistently surpasses state-of-the-art distillation methods. Moreover, our method demonstrates substantial effectiveness when distilling Vision Transformers, underscoring its versatility across diverse network architectures. These findings highlight the significant potential of logits to advance knowledge distillation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
