MetaDistiller：通过元学习自顶向下蒸馏实现网络自提升
MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
摘要：
知识蒸馏（KD）是学习紧凑模型的最流行的方法之一。然而，它仍然遭受高的时间和计算资源的顺序训练流水线造成的要求。此外，由于兼容性的差距，来自较深模型的软目标通常不能作为较浅模型的良好线索。在这项工作中，我们同时考虑这两个问题，具体地说，我们提出了更好的软目标，具有更高的兼容性，可以通过使用标签生成器融合的特征maps从更深的阶段，在自顶向下的方式，我们可以使用元学习技术来优化这个标签生成器。利用从模型的中间特征图中学习到的软目标，我们可以实现比现有技术更好的网络自增强。实验在两个标准分类基准CIFAR-100和ILSVRC 2012上进行。我们测试了各种网络架构，以显示我们的MetaDistiller的通用性。在两个数据集上的实验结果表明了该方法的有效性。
英文摘要：
Knowledge Distillation (KD) has been one of the most popu-lar methods to learn a compact model. However, it still suffers from highdemand in time and computational resources caused by sequential train-ing pipeline. Furthermore, the soft targets from deeper models do notoften serve as good cues for the shallower models due to the gap of com-patibility. In this work, we consider these two problems at the same time.Specifically, we propose that better soft targets with higher compatibil-ity can be generated by using a label generator to fuse the feature mapsfrom deeper stages in a top-down manner, and we can employ the meta-learning technique to optimize this label generator. Utilizing the softtargets learned from the intermediate feature maps of the model, we canachieve better self-boosting of the network in comparison with the state-of-the-art. The experiments are conducted on two standard classificationbenchmarks, namely CIFAR-100 and ILSVRC2012. We test various net-work architectures to show the generalizability of our MetaDistiller. Theexperiments results on two datasets strongly demonstrate the effective-ness of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
