Speculative Knowledge Distillation：通过交错采样弥合师生差距
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling
AI总结：
提出 Speculative Knowledge Distillation（SKD），通过学生与教师模型协作实时生成高质量训练数据，自适应转移高质量知识，在多种文本生成任务中优于现有 KD 方法。
摘要：
知识蒸馏（KD）的最新进展使得较小的学生模型能够接近较大教师模型的性能。然而，监督式 KD 和 on-policy KD 等流行方法在实际场景中会受到师生之间知识差距的不利影响。监督式 KD 在使用静态数据集进行训练与对学生生成的最终输出进行推理之间存在分布不匹配的问题。相反，使用学生生成样本进行训练的 on-policy KD 可能会受到低质量训练样本的影响，教师模型对这些样本不熟悉，从而导致不准确的教师反馈。为了解决这些局限性，我们引入了 Speculative Knowledge Distillation（SKD），这是一种新颖的方法，利用学生模型和教师模型之间的合作，实时生成高质量训练数据，同时与学生的推理时分布保持一致。在 SKD 中，学生提出 token，教师根据自身的分布替换排名较低的 token，自适应地转移高质量知识。我们在各种文本生成任务上评估了 SKD，包括翻译、摘要、数学和指令遵循，并表明 SKD 在不同领域、数据规模和模型初始化策略中始终优于现有的 KD 方法。
英文摘要：
Recent advances in knowledge distillation (KD) have enabled smaller student models to approach the performance of larger teacher models. However, popular methods such as supervised KD and on-policy KD, are adversely impacted by the knowledge gaps between teacher-student in practical scenarios. Supervised KD suffers from a distribution mismatch between training with a static dataset and inference over final student-generated outputs. Conversely, on-policy KD, which uses student-generated samples for training, can suffer from low-quality training examples with which teacher models are not familiar, resulting in inaccurate teacher feedback. To address these limitations, we introduce Speculative Knowledge Distillation (SKD), a novel approach that leverages cooperation between student and teacher models to generate high-quality training data on-the-fly while aligning with the student's inference-time distribution. In SKD, the student proposes tokens, and the teacher replaces poorly ranked ones based on its own distribution, transferring high-quality knowledge adaptively. We evaluate SKD on various text generation tasks, including translation, summarization, math, and instruction following, and show that SKD consistently outperforms existing KD methods across different domains, data sizes, and model initialization strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
