通过自学来提炼自我：通过自我知识蒸馏进行特征提炼
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation
摘要：
知识蒸馏是一种将知识从预训练的复杂教师模型转移到学生模型的方法，因此较小的网络可以在部署阶段取代大型教师网络。为了减少训练大型教师模型的必要性，最近的文献引入了自知识蒸馏，其在没有预先训练的教师网络的情况下逐步训练学生网络以提取其自身的知识。虽然自知识蒸馏主要分为基于数据增强的方法和基于辅助网络的方法，但数据增强方法在增强过程中丢失了其局部信息，这阻碍了其对诸如语义分割的各种视觉任务的适用性。此外，这些知识蒸馏方法不接收在目标检测和语义分割社区中普遍存在的细化的特征图。本文提出了一种新的自知识提取方法--基于自知识蒸馏的特征精炼（FRSKD），它利用辅助自教师网络向分类器网络传递精炼后的知识。我们提出的方法，FRSKD，可以利用软标签和特征图蒸馏的自我知识蒸馏。因此，FRSKD可以应用于分类，语义分割，强调保持局部信息。我们证明了FRSKD的有效性，列举其在不同的任务和基准数据集的性能改进。实现的代码可在https://github.com/MingiJi/FRSKD上获得。
英文摘要：
Knowledge distillation is a method of transferring the knowledge from a pretrained complex teacher model to a student model, so a smaller network can replace a large teacher network at the deployment stage. To reduce the necessity of training a large teacher model, the recent literatures introduced a self-knowledge distillation, which trains a student network progressively to distill its own knowledge without a pretrained teacher network. While Self-knowledge distillation is largely divided into a data augmentation based approach and an auxiliary network based approach, the data augmentation approach looses its local information in the augmentation process, which hinders its applicability to diverse vision tasks, such as semantic segmentation. Moreover, these knowledge distillation approaches do not receive the refined feature maps, which are prevalent in the object detection and semantic segmentation community. This paper proposes a novel self-knowledge distillation method, Feature Refinement via Self-Knowledge Distillation (FRSKD), which utilizes an auxiliary self-teacher network to transfer a refined knowledge for the classifier network. Our proposed method, FRSKD, can utilize both soft label and feature-map distillations for the self-knowledge distillation. Therefore, FRSKD can be applied to classification, and semantic segmentation, which emphasize preserving the local information. We demonstrate the effectiveness of FRSKD by enumerating its performance improvements in diverse tasks and benchmark datasets. The implemented code is available at https://github.com/MingiJi/FRSKD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
