基于多源信息融合的轻量级自知识蒸馏
Lightweight Self-Knowledge Distillation with Multi-source Information Fusion
AI总结：
本文提出一种轻量级自知识蒸馏框架，通过引入带反向引导的蒸馏（DRG）和带形状正则化的蒸馏（DSR）方法，利用多源信息构建更丰富的教师模型，在保持计算高效的同时显著优于基线和现有SKD方法。
摘要：
知识蒸馏（KD）是一种在神经网络模型之间传递知识的强大技术，它利用预训练的教师模型来促进目标学生模型的训练。然而，并不总是能保证获得合适的教师模型。为了应对这一挑战，自知识蒸馏（SKD）试图从模型自身构建教师模型。现有的SKD方法在模型的中间层添加辅助分类器（AC），或使用历史模型以及同一类别内具有不同输入数据的模型。但是，这些方法计算成本高昂，并且仅捕获数据的时间维度和类别维度特征。在本文中，我们提出了一个轻量级的SKD框架，利用多源信息来构建信息更丰富的教师模型。具体而言，我们引入了带反向引导的蒸馏（DRG）方法，该方法考虑模型提取的不同层次信息（包括输入数据的边缘、形状和细节），以构建信息更丰富的教师模型。此外，我们设计了带形状正则化的蒸馏（DSR）方法，以确保所有数据的排序模型输出形状一致。我们通过各种数据集和模型上的综合实验验证了所提出的DRG、DSR及其组合的性能。结果表明，所提出的方法优于基线（高达2.87%）和最先进的SKD方法（高达1.15%），同时具备计算高效性和鲁棒性。代码可在 https://github.com/xucong-parsifal/LightSKD 获取。
英文摘要：
Knowledge Distillation (KD) is a powerful technique for transferring knowledge between neural network models, where a pre-trained teacher model is used to facilitate the training of the target student model. However, the availability of a suitable teacher model is not always guaranteed. To address this challenge, Self-Knowledge Distillation (SKD) attempts to construct a teacher model from itself. Existing SKD methods add Auxiliary Classifiers (AC) to intermediate layers of the model or use the history models and models with different input data within the same class. However, these methods are computationally expensive and only capture time-wise and class-wise features of data. In this paper, we propose a lightweight SKD framework that utilizes multi-source information to construct a more informative teacher. Specifically, we introduce a Distillation with Reverse Guidance (DRG) method that considers different levels of information extracted by the model, including edge, shape, and detail of the input data, to construct a more informative teacher. Additionally, we design a Distillation with Shape-wise Regularization (DSR) method that ensures a consistent shape of ranked model output for all data. We validate the performance of the proposed DRG, DSR, and their combination through comprehensive experiments on various datasets and models. Our results demonstrate the superiority of the proposed methods over baselines (up to 2.87%) and state-of-the-art SKD methods (up to 1.15%), while being computationally efficient and robust. The code is available at https://github.com/xucong-parsifal/LightSKD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
