发表机构
Tongji University; Xiaohongshu; Shanghai Jiao Tong University; Hunan University（同济大学; 小红书; 上海交通大学; 湖南大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出成功参照的在线策略蒸馏（SR-OPD），利用学生成功轨迹选择失败轨迹进行教师监督，在固定预算下大幅降低教师令牌消耗并保持推理性能。
AI 中文摘要
在线策略蒸馏（OPD）将学生生成的轨迹与教师提供的密集令牌级监督相结合，但对每条轨迹提供此类监督需要大量的教师计算。我们提出了成功参照的在线策略蒸馏（SR-OPD），通过选择哪些提示和轨迹接受教师监督来降低这一成本。当学生对同一提示同时产生成功和失败的轨迹时，成功轨迹可以作为选择失败轨迹的自然参照。因此，SR-OPD聚焦于此类提示，并优先选择隐藏状态轨迹与成功参照表现出持续分歧的失败轨迹，同时考虑估计的教师输入成本。在三个师生对和六个数学推理基准上，SR-OPD在单遍设置中仅使用Vanilla OPD所需教师输入令牌的3.46%-5.02%，同时保持相当的性能。在与Vanilla OPD教师输入预算的5%匹配的受控设置下，进一步实验支持两个关键设计选择：将监督聚焦于同时具有成功和失败轨迹的提示，以及使用成功轨迹指导失败选择。这些结果表明，在固定的教师输入预算下，学生自身成功的行为可以作为分配教师监督的有用参照。
英文摘要
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.
Comments18 pages, 2 figures. Submitted to ICLR 2027
