arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.03263cs.LGcs.AI
收藏

蒸馏作为小模型预训练替代方案的惊人有效性

On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models

University of Illinois, Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Sean Farhat, Deming Chen

AI总结:

本文提出用知识蒸馏替代小模型预训练,通过将蒸馏归约为对比学习并利用生成模型增强数据,在保持性能的同时使训练速度提升高达94%。

摘要:

在本文中,我们提出小模型可能无需承担预训练的成本即可获得其收益。相反,它们可以在惊人程度上利用现代巨型模型取得的卓越成果。我们观察到,当小模型在某个任务上从预训练教师模型进行蒸馏时,其性能可以达到或超过先预训练再在该任务上微调所能达到的水平。为了使这一现象易于利用,我们建立了一种将知识蒸馏归约为现代对比学习的联系,从而打开了两扇门:(1)差异极大的模型架构组合可以用于蒸馏;(2)大多数根植于噪声对比估计(Noise Contrastive Estimation)理论的对比学习算法都可以被轻松应用和使用。我们使用来自开源模型库的预训练教师模型、基于Transformer和卷积的模型组合,以及一种新颖的蒸馏算法来展示这一范式,该算法将Wang & Isola(2020)提出的对比学习对齐/均匀性(Alignment/Uniformity)视角转化为蒸馏目标。我们选择这种对比学习形式是因为其计算成本低,这也是本工作的一个总体主题。我们还观察到,如果任务数据有限,这一现象往往不会发生。然而,这可以通过利用另一个受规模启发的进展来缓解:用于数据集增强的大型预训练生成模型。我们同样使用开源模型,并且我们简单的提示就足以提升小模型的性能。因此,我们强调了一种小模型训练方法,其速度比标准预训练范式快达94%,且不牺牲性能。对于因规模过大而无法充分利用现代基础数据集来训练小模型的实践者,我们相信我们的工作保持了这条道路的开放。

英文摘要:

In this paper, we propose that small models may not need to absorb the cost of pre-training to reap its benefits. Instead, they can capitalize on the astonishing results achieved by modern, enormous models to a surprising degree. We observe that, when distilled on a task from a pre-trained teacher model, a small model can achieve or surpass the performance it would achieve if it was pre-trained then finetuned on that task. To allow this phenomenon to be easily leveraged, we establish a connection reducing knowledge distillation to modern contrastive learning, opening two doors: (1) vastly different model architecture pairings can work for the distillation, and (2) most contrastive learning algorithms rooted in the theory of Noise Contrastive Estimation can be easily applied and used. We demonstrate this paradigm using pre-trained teacher models from open-source model hubs, Transformer and convolution based model combinations, and a novel distillation algorithm that massages the Alignment/Uniformity perspective of contrastive learning by Wang & Isola (2020) into a distillation objective. We choose this flavor of contrastive learning due to its low computational cost, an overarching theme of this work. We also observe that this phenomenon tends not to occur if the task is data-limited. However, this can be alleviated by leveraging yet another scale-inspired development: large, pre-trained generative models for dataset augmentation. Again, we use an open-source model, and our rudimentary prompts are sufficient to boost the small model`s performance. Thus, we highlight a training method for small models that is up to 94% faster than the standard pre-training paradigm without sacrificing performance. For practitioners discouraged from fully utilizing modern foundation datasets for their small models due to the prohibitive scale, we believe our work keeps that door open.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑