前缀教导,后缀消退:强到弱在线策略蒸馏中的局部可教性崩溃
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
AI总结:
本文发现强到弱在线策略蒸馏中,教师反馈在生成轨迹的后缀部分缺乏局部对比度,导致“局部可教性崩溃”,并提出基于变化点检测的截断规则来优化监督区域,实验表明该方法优于全轨迹蒸馏。
摘要:
在线策略蒸馏(OPD)利用来自更强教师的密集反馈,在学生模型自身的生成轨迹上训练学生模型。先前文献表明,只要教师反馈可用,监督完整响应令牌序列应能单调提升性能。然而,我们证明这一假设在强到弱OPD设置中有时不成立。虽然生成轨迹的后缀部分可能仍存在非零的师生优势,但它们通常缺乏使密集反馈有效优先学生学习所需的局部对比度。我们将这种失败模式称为局部可教性崩溃。由此得出的原则很简单:监督应集中在教师反馈仍具有判别性的轨迹区域,而非均匀覆盖整个响应。我们通过一种轨迹特定的释放规则来操作这一原则。该规则测量教师相对于学生前K个候选集的边际,将该边际在NLTK分词的句子片段上聚合,并在检测到BIC风格的下行变化点时截断密集OPD监督。使用Qwen3模型系列在强到弱蒸馏任务上的实验结果表明,该释放规则在不同学生规模下的五个域内基准上始终优于标准全轨迹OPD。此外,与基线蒸馏方法相比,我们的方法在域外任务上更好地保留了模型能力。这些结果表明,有效的强到弱OPD需要评估教师指导的可用性及其局部效用,确保生成的反馈保持可教性。
英文摘要:
On-policy distillation (OPD) trains a student model on its own rollouts using dense feedback from a stronger teacher. Prior literature suggests that, provided teacher feedback is available, supervising the full sequence of response tokens should monotonically improve performance. However, we demonstrate that this assumption sometimes fails to hold in strong-to-weak OPD settings. While later segments of a generated trajectory may still exhibit a non-zero teacher-student advantage, they frequently lack the local contrast that makes dense feedback effective for prioritizing student learning. We term this failure mode local teachability collapse. The resulting principle is straightforward: supervision should concentrate on trajectory regions where the teacher's feedback remains discriminative, rather than uniformly covering the entire response. We operationalize this principle through a trajectory-specific release rule. This rule measures the teacher's margin over the student's top-$K$ candidate set, aggregates this margin across NLTK-tokenized sentence segments, and truncates dense OPD supervision upon detecting a BIC-style downward change point. Experimental results across strong-to-weak distillation tasks using the Qwen3 model family indicate that this release rule consistently outperforms standard full-trajectory OPD across five in-domain benchmarks at various student scales. Furthermore, compared to baseline distillation methods, our approach better preserves model capabilities on out-of-domain task. These results suggest that effective strong-to-weak OPD requires evaluating not only the availability of teacher guidance but also its local utility, ensuring that the generated feedback remains teachable.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。