当裁剪逆转修正：逐点前向KL在线策略自蒸馏的失败动力学
When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation
- Washington University in St. Louis（圣路易斯华盛顿大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究揭示在线策略自蒸馏中逐点裁剪前向KL目标会导致响应重复加剧，证明裁剪目标可能逆转修正方向，将学生概率推离教师模型，并建议谨慎使用该裁剪技术。
AI中文摘要：
在线策略自蒸馏（OPSD）通过利用同一模型基于特权信息提供的反馈，在模型自身生成的响应上训练学生模型。在数学推理任务中，原始的OPSD研究发现，风格化标记在训练信号中可能主导数学相关标记，并且对前向KL目标进行逐点裁剪可以稳定训练。逐点裁剪在词汇表上求和之前，将每个词汇级别的前向KL项限制在固定阈值内。后续研究采用了这种裁剪方法，但其对训练的影响尚未被直接检验。在仅因是否应用裁剪而有所不同的匹配训练运行中，我们观察到，与未裁剪的运行相比，裁剪后的运行产生了明显更多的重复，这些重复持续到响应结束。我们将这一失败追溯到裁剪目标。我们证明了裁剪目标可能无法将学生模型向教师模型修正，反而可能将裁剪和未裁剪的标记概率推离教师模型。我们的训练运行与此分析一致：在重复内部，裁剪后的学生模型比教师模型更少地赋予退出重复的概率，而更多地赋予继续重复的概率，而未裁剪的运行则保持接近教师模型。
英文摘要：
On-policy self-distillation (OPSD) trains a student on its own generated responses using feedback from the same model conditioned on privileged information. On mathematical reasoning, the original OPSD study finds that stylistic tokens can dominate the training signal over math-related tokens, and that pointwise clipping of the forward KL objective stabilizes training. Pointwise clipping caps each vocabulary-wise forward KL term at a fixed threshold before summing over the vocabulary. Follow-up studies have adopted this clipping, but its effect on training has not been directly examined. In matched training runs differing only in whether clipping is applied, we observe that clipped runs produce substantially more repetitions that persist to the end of the response than their unclipped counterparts. We trace this failure to the clipped objective. We prove that the clipped objective can fail to correct the student toward the teacher and can instead push clipped and unclipped token probabilities away from its teacher. Our training runs agree with this analysis: inside repetitions, the clipped student places less probability than its teacher on leaving the repetition, and more on continuing it, whereas the unclipped runs stay close to their teachers.
