arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.16844cs.CV
收藏

精馏教学中师生知识离散性的修正

Fixing the Teacher-Student Knowledge Discrepancy in Distillation

SenseTime Research(商汤科技研究院);CUHK-SenseTime Joint Laboratory, The Chinese University of Hong Kong(香港中文大学-商汤科技联合实验室,香港中文大学)

Jiangfan Han, Mengya Gao, Yujie Wang, Quanquan Li, Hongsheng Li, Xiaogang Wang

摘要:

在较大的教师网络的指导下训练较小的学生网络是促进学生成绩的有效途径。以往的知识蒸馏方法中,尽管教师和学生的类型不同,但用于蒸馏的引导知识对于不同的师生对总是保持不变的。然而,我们发现,具有不同网络或从不同初始化训练的教师和学生模型在不同通道之间可能具有不同的特征表示。(e.g.不同类别的高激活通道)。我们将这种渠道的不协调表征称为蒸馏过程中的师生知识差异。忽视教师和学生模型的知识差异问题,将使学生向教师学习更加困难。针对这一问题,本文提出了一种新的基于学生的蒸馏方法--知识一致性蒸馏,该方法使教师的知识与学生的知识更加一致,并为不同的学生网络提供最适合的知识进行蒸馏。在不同数据集(CIFAR 100,ImageNet,COCO)和任务(图像分类,目标检测)上的实验揭示了教师和学生之间普遍存在的知识差异问题,证明了该方法的有效性。我们的方法非常灵活,可以很容易地与其他最先进的方法相结合。

英文摘要:

Training a small student network with the guidance of a larger teacher network is an effective way to promote the performance of the student. Despite the different types, the guided knowledge used to distill is always kept unchanged for different teacher and student pairs in previous knowledge distillation methods. However, we find that teacher and student models with different networks or trained from different initialization could have distinct feature representations among different channels. (e.g. the high activated channel for different categories). We name this incongruous representation of channels as teacher-student knowledge discrepancy in the distillation process. Ignoring the knowledge discrepancy problem of teacher and student models will make the learning of student from teacher more difficult. To solve this problem, in this paper, we propose a novel student-dependent distillation method, knowledge consistent distillation, which makes teacher's knowledge more consistent with the student and provides the best suitable knowledge to different student networks for distillation. Extensive experiments on different datasets (CIFAR100, ImageNet, COCO) and tasks (image classification, object detection) reveal the widely existing knowledge discrepancy problem between teachers and students and demonstrate the effectiveness of our proposed method. Our method is very flexible that can be easily combined with other state-of-the-art approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑