TISD：带轨迹干预的在线策略自蒸馏
TISD: On-Policy Self-Distillation with Trajectory Intervention
- KAIST（韩国科学技术院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对在线策略自蒸馏中教师分歧仅作局部修复的局限，提出TISD算法，强制教师选择分支动作并蒸馏完整轨迹，在编码和科学任务上分别提升Avg@4达1.2点、Avg@128达0.8点。
AI中文摘要：
在线策略自蒸馏（OPSD）提供了密集的教师目标，但仅沿学生采样的轨迹对其进行评估。当特权教师在访问的前缀处倾向于替代动作时，OPSD可以为分支决策提供目标，但除非学生采样该动作，否则无法监督由该动作引发的后续上下文。这造成了训练时的数据收集瓶颈，并暗示了师生分歧的不同作用：提出轨迹分支而非识别足够的局部修复。我们的诊断框架使用受控的令牌干预揭示，在峰值分歧处教师偏好的令牌可以提高学生继续生成的成功率，而其局部纠正价值有限。受此发现启发，我们引入了一种简单的分支-重新生成-蒸馏算法，即轨迹干预自蒸馏（TISD）。TISD强制采取教师选择的分支动作，将后缀生成交还给学生，并在特权上下文条件下的教师指导下蒸馏完整轨迹。在编码模型中，TISD相比SDPO将平均Avg@4提高了1.2个百分点。在科学领域，在等步数预算下平均Avg@128提高了0.8点，在等时间预算下提高了0.3点。这些结果支持教师引导的分支作为暴露有用后续上下文以进行自蒸馏的方式。
英文摘要：
On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher-student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation.
