发表机构
FirstPrinciples（FirstPrinciples）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究针对Qwen3模型开展OPSD实验，分析OPSD的性能提升源于参考信息还是基础模型的推理行为，发现性能提升与参考信息的契合度无可靠关联。
AI 中文摘要
策略内自蒸馏（OPSD）利用特权参考信息，通过同一模型的token级监督训练学生模型，学生模型以自身响应为学习对象。本研究探究OPSD带来的性能提升，是因为学生模型学习到了参考信息，还是源于对基础模型中已存在的推理行为的恢复。我们使用Qwen3模型（规模从17亿参数到80亿参数），在科学与数学数据集上开展OPSD实验。我们的分析框架将参考信息带来的监督与无参考信息时教师模型提供的监督分离开来，并分别衡量两者与学生模型预测变化的契合度。正确的参考信息在教师生成模式、模型规模和训练数据集上并未带来一致的性能提升；学生模型在无正确参考信息时也能实现性能提升，且其他问题的解决方案在多个数学推理基准测试中表现优于正确解决方案。学生模型的预测与基础模型的推理行为契合度，高于与参考信息带来的监督的契合度，但从其他问题构建的控制项可同时复制这两种契合度。此外，正确参考信息带来的更强契合度，并未可靠地与参考信息带来的更大性能提升相重合。因此，仅靠性能提升和分布契合度无法确定特权参考信息在OPSD中对学生模型学习的贡献方式。
英文摘要
On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information. We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model. We perform OPSD experiments on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B. Our analysis framework separates the supervision induced by the reference from the supervision provided by the teacher without the reference and measures how each aligns with changes in the student's predictions. The correct reference does not provide a consistent performance benefit across teacher generation modes, model sizes, and training datasets. Students can improve without the correct reference, and a solution from another problem can outperform the correct solution on several mathematical reasoning benchmarks. The student's predictions align more strongly with the base model's thinking behavior than with the supervision induced by the reference, but controls constructed from other problems reproduce much of both alignments. Moreover, stronger alignment attributable to the correct reference does not reliably coincide with a greater performance benefit from the reference. Performance gains and distributional alignment alone therefore cannot determine how privileged reference information contributes to student learning in OPSD.
Comments21 pages, 7 figures
