arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.28048cs.CLcs.AIcs.LG
收藏

在线策略迁移蒸馏

On-Policy Transition Distillation

The Hong Kong Polytechnic University(香港理工大学);Huawei Technologies Ltd.(华为技术有限公司)

Runyang You, Yongqi Li, Tiezheng Yu, Wenjie Li

AI总结:

本研究提出在线策略迁移蒸馏(OPTD),通过对齐教师与学生的逐层对数概率增量补充最终层分布匹配,在多基准蒸馏中恢复约90%的教师-学生能力差距,性能优于在线策略蒸馏(OPD)。

摘要:

在线策略蒸馏(OPD)通过匹配教师模型与学生模型在学生生成的rollout上的最终层分布,将专家能力整合到单个学生模型中。本研究探究层间的深度预测迁移(即各层预测的变化)是否能提供额外的任务相关监督。诊断研究发现,中间迁移中教师与学生的一致性越强,问题内的答案正确率越高,正确响应在所有测量的中间层对中表现出更低的平均迁移KL。这些观察结果启发了在线策略迁移蒸馏(OPTD),该方法通过对齐教师与学生在学生生成rollout上的逐层对数概率增量,补充了最终层分布匹配。在涵盖数学、编码和指令遵循的三个骨干网络及八个基准上的评估显示,OPTD在双域和三域蒸馏中均取得最佳或并列最佳的整体性能。在三域设置中,OPTD恢复了约90%的整体教师-学生能力差距,而OPD仅恢复约75%。

英文摘要:

On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts. We investigate whether depthwise prediction transitions, the changes in predictions between layers, can provide additional task-relevant supervision. Our diagnostic study finds that stronger teacher--student agreement in intermediate transitions is associated with higher answer correctness within questions, with correct responses exhibiting lower mean transition KL across all measured intermediate layer pairs. These observations motivate On-Policy Transition Distillation (OPTD), which augments final-layer distribution matching by aligning teacher and student layer-to-layer log-probability increments on student-generated rollouts. Evaluations across three backbones and eight benchmarks spanning mathematics, coding, and instruction following show that OPTD achieves the best or tied-best overall performance in both dual- and three-domain distillation. In the three-domain setting, OPTD recovers $\approx 90\%$ of the overall teacher--student capability gap, compared with $\approx 75\%$ for OPD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑