arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.06849cs.LGcs.AIstat.ML
收藏

蒸馏双下降

Distilling Double Descent

Google Research(谷歌研究院)

Andrew Cotter, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, Sashank J. Reddi, Yichen Zhou

摘要:

蒸馏是一种基于由单独的“教师”模型标记的示例来训练“学生”模型的技术,该模型本身是在标记的数据集上训练的。对于为什么蒸馏“有效”的最常见的解释是基于这样的假设,即学生被提供了来自教师模型的\emph{soft}标签、\eg概率或置信度。在这项工作中,我们表明,即使教师模型是高度过度参数化的,并提供\emph{hard}标签,使用一个非常大的保持了未标记的数据集训练学生模型可以导致一个模型,优于更“传统”的方法。我们对这一现象的解释是基于最近对“双下降“的研究。已经观察到,一旦模型的复杂度大致超过了记忆训练数据所需的量,增加复杂度\emph{further}可能会违反直觉地导致\emph{better}泛化。研究人员已经确定了几种发生这种情况的环境,而其他人则进行了各种尝试来解释它(到目前为止,只有部分成功)。相比之下,我们避免了这些问题,而是通过证明高度过度参数化的教师可以通过双下降避免过度拟合来寻求\emph{exploit}这种现象,而在由该教师标记的较大独立数据集上训练的学生将避免由于其训练集的大小而导致的过度拟合。

英文摘要:

Distillation is the technique of training a "student" model based on examples that are labeled by a separate "teacher" model, which itself is trained on a labeled dataset. The most common explanations for why distillation "works" are predicated on the assumption that student is provided with \emph{soft} labels, \eg probabilities or confidences, from the teacher model. In this work, we show, that, even when the teacher model is highly overparameterized, and provides \emph{hard} labels, using a very large held-out unlabeled dataset to train the student model can result in a model that outperforms more "traditional" approaches. Our explanation for this phenomenon is based on recent work on "double descent". It has been observed that, once a model's complexity roughly exceeds the amount required to memorize the training data, increasing the complexity \emph{further} can, counterintuitively, result in \emph{better} generalization. Researchers have identified several settings in which it takes place, while others have made various attempts to explain it (thus far, with only partial success). In contrast, we avoid these questions, and instead seek to \emph{exploit} this phenomenon by demonstrating that a highly-overparameterized teacher can avoid overfitting via double descent, while a student trained on a larger independent dataset labeled by this teacher will avoid overfitting due to the size of its training set.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑