发表机构
Tencent（腾讯）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对在线策略蒸馏中教师知识分配不明确的问题，提出RouteOPD方法，通过概率输运显式建模师生分歧，在多个基准上优于现有方法。
AI 中文摘要
在线策略蒸馏（OPD）在由学生生成的轨迹上传递教师知识，但高效的采样目标将教师分布简化为单个令牌上的标量信用。这种信用指示了某个令牌应增加还是减少概率，却未指定相应的重新分配方式。我们将OPD重新表述为教师引导的概率输运，并提出RouteOPD（路由式在线策略蒸馏），它将局部的师生分歧分解为学生过剩源和教师不足目的地，并将它们耦合成显式的输运对。RouteOPD优化成对的对数几率，朝向由有界教师势获得的联合可实现目标，同时根据教师需求的集中度调整输运预算。该公式将更新引导至教师偏好的目的地，并在单个输运算子内控制其幅度。在四个师生设置和四个数学推理基准上的实验表明，RouteOPD始终优于采样的反向KL OPD，其改进伴随着更高的路由保真度和更低的背景泄漏。这些结果证明了在在线策略蒸馏中显式建模概率输运的有效性。
英文摘要
On-policy distillation (OPD) transfers teacher knowledge on student-generated trajectories, but efficient sampled objectives reduce the teacher distribution to scalar credit on individual tokens. Such credit indicates whether a token should gain or lose probability, yet leaves the corresponding redistribution unspecified. We recast OPD as teacher-guided probability transport and propose RouteOPD (Routed On-Policy Distillation), which decomposes local teacher--student disagreement into student-excess sources and teacher-deficit destinations and couples them into explicit transport pairs. RouteOPD optimizes pairwise log-odds toward jointly realizable targets obtained from a bounded teacher potential, while adapting the transport budget to the concentration of teacher demand. This formulation directs updates toward teacher-preferred destinations and controls their magnitude within a single transport operator. Experiments across four teacher--student settings and four mathematical-reasoning benchmarks demonstrate that RouteOPD consistently outperforms sampled reverse-KL OPD, with improvements accompanied by higher routing fidelity and lower background leakage. These results demonstrate the effectiveness of explicitly modeling probability transport in on-policy distillation.
