arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.00051cs.LGstat.ML
收藏

Extracurricular Learning:超越经验分布的知识转移

Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution

Apple(苹果公司);UCSD(加州大学圣地亚哥分校)

Hadi Pouransari, Mojan Javaheripi, Vinay Sharma, Oncel Tuzel

摘要:

知识蒸馏已被用于将由复杂模型(教师)学习的知识转移到更简单的模型(学生)。该技术被广泛用于压缩模型复杂性。然而,在大多数应用中,压缩的学生模型与其教师存在精度差距。我们提出了课外学习,一种新的知识蒸馏方法,通过(1)对学生和教师的输出分布进行建模,(2)从底层数据分布的近似值中采样示例,以及(3)在包括不确定样本的扩展集上匹配学生和教师的输出分布,来弥合这一差距。我们对回归和分类任务进行了严格的评估,结果表明,与标准的知识蒸馏相比,课外学习将差距缩小了46%至68%。与各种最近的神经网络架构的基于经验风险最小化的训练相比,这导致了重大的准确性改进:MPIIGaze数据集上的回归误差减少了16%,CIFAR 100数据集上的top-1分类准确性提高了+3.4%到+9.1%,ImageNet数据集上的top-1提高了+2.9%。

英文摘要:

Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model complexity. However, in most applications the compressed student model suffers from an accuracy gap with its teacher. We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output distributions; (2) sampling examples from an approximation to the underlying data distribution; and (3) matching student and teacher output distributions over this extended set including uncertain samples. We conduct rigorous evaluations on regression and classification tasks and show that compared to the standard knowledge distillation, extracurricular learning reduces the gap by 46% to 68%. This leads to major accuracy improvements compared to the empirical risk minimization-based training for various recent neural network architectures: 16% regression error reduction on the MPIIGaze dataset, +3.4% to +9.1% improvement in top-1 classification accuracy on the CIFAR100 dataset, and +2.9% top-1 improvement on the ImageNet dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑