解构自蒸馏：测量与建模获取与保持

中文摘要 AI 辅助

带特权上下文的自蒸馏通过让模型在给定参考响应后，逐词元地教导其无上下文副本，从而从演示中适应语言模型。我们的分类法揭示了现有方法在三个纠缠的轴上有所不同：(i) 展开源（学生或教师），(ii) 教师耦合（冻结，或以某种耦合率对学生进行指数移动平均），以及 (iii) KL方向（反向或正向），然而这些轴通常以固定组合进行研究，并导致了相互矛盾的结论。我们形式化了一个统一框架，以涵盖所有自蒸馏方法与经典监督微调：我们在Qwen2.5-7B和Ministral-3-3B上，在普通和矛盾任务中训练三个轴的每一种组合，总计1,200次适应运行，以系统研究上述轴的影响。我们提出了一个受控的相同目标模型，以解释由此产生的获取-保持权衡。我们发现：(i) 展开源主要在任务与预训练行为相矛盾时起作用：在那里，教师展开将获取提升到远高于学生展开所能达到的水平，而保持几乎没有变化；(ii) 教师耦合在每个任务上对获取的改变最大：获取随耦合率上升，然后超过任务特定比率后下降；(iii) 切换KL方向在一个模型中损失保持，但在另一个模型中则不然，因此先调整哪个轴取决于模型。受控模型重现了这三种趋势。

英文摘要

Self-distillation with privileged context adapts a language model from demonstrations by letting the model, once conditioned on a reference response, teach its context-free copy token by token. Our taxonomy reveals existing methods differ along three entangled axes: (i) the rollout source (student or teacher), (ii) the teacher coupling (frozen, or an exponential moving average of the student at some coupling rate) and (iii) the KL direction (reverse or forward), yet these axes are usually studied in fixed combinations and have led to conflicting conclusions. We formalize a unifying framework to encompass all self-distillation methods vs classic supervised fine-tuning: we train every combination of the three axes, on Qwen2.5-7B and Ministral-3-3B across ordinary and contradictory tasks, totaling 1,200 adaptation runs, to systematically investigate the impact of the above axes. We propose a controlled model of the same objective to explain the resulting acquisition-retention trade-offs. We find that (i) the rollout source matters mostly where the task contradicts the pretrained behavior: there teacher rollouts raise acquisition well above what student rollouts achieve, with almost no change in retention; (ii) the teacher coupling changes acquisition most, on every task: acquisition rises with the coupling rate, then falls past a task-specific rate; (iii) switching the KL direction costs retention in one model but not the other so which axis to tune first depends on the model. The controlled model reproduces the three trends.

发表机构

• Huawei Technologies Co., Ltd.（华为技术有限公司）

机构由 AI 辅助整理，请以论文原文为准。