发表机构
University of Hong Kong; Zhejiang University; Dalian University of Technology（香港大学; 浙江大学; 大连理工大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出前缀恢复指标，通过测量学生完成教师部分解决方案的能力，结合梯度冲突分析，发现反向KL蒸馏在低容量模型中效果最佳，为分布级蒸馏选择提供实用诊断。
AI 中文摘要
一个正确的教师解决方案，当接收的学生能够继续其推理时，便成为有用的监督信号。我们通过前缀恢复来衡量这种兼容性：在揭示已验证解决方案的25%、50%或75%后，我们测试学生是否能正确完成它。我们将恢复与全词汇表上交叉熵和反向KL梯度之间的余弦冲突联系起来。在从0.6B到8B参数的相邻Qwen3教师-学生对中，反向KL蒸馏为两个低于2B参数的学生带来了最一致的数学和代码改进。在固定的1,000条轨迹队列上，随着学生规模从0.6B增加到4B，平均前缀恢复率从71.0%上升到91.9%，而稳健-脆弱恢复差距从46.0个百分点缩小到14.4个百分点。当教师固定为8B时，冲突分离度从0.993下降到0.233。一项独立的目标干预发现，反向KL在脆弱轨迹上的救援效果最大。这三项测量定位了相同的容量依赖转移机制：当正确轨迹仍然不均匀可恢复时，分布匹配具有最大的提升空间。前缀恢复为选择昂贵的分布级蒸馏提供了一种实用的诊断方法。
英文摘要
A correct teacher solution becomes useful supervision when the receiving student can continue its reasoning. We measure this compatibility with prefix recovery: after revealing 25%, 50%, or 75% of a verified solution, we test whether the student completes it correctly. We connect recovery to the cosine conflict between cross-entropy and reverse-KL gradients over the full vocabulary. Across adjacent Qwen3 teacher-student pairs from 0.6B to 8B parameters, reverse-KL distillation delivers its most consistent mathematical and code improvements for the two students below 2B parameters. On a fixed cohort of 1,000 trajectories, average prefix recovery rises from 71.0% to 91.9% as student size increases from 0.6B to 4B, and the robust-fragile recovery gap contracts from 46.0 to 14.4 percentage points. With the teacher fixed at 8B, conflict separation falls from 0.993 to 0.233. An independent objective intervention finds the largest reverse-KL rescue on fragile trajectories. The three measurements locate the same capacity-dependent transfer regime: distribution matching has the greatest headroom when correct traces remain unevenly recoverable. Prefix recovery provides a practical diagnostic for selecting costly distribution-level distillation.
