arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.38637cs.LGcs.AI
收藏

基于教师进化的在线策略自蒸馏

On-Policy Self-Distillation with Teacher Evolution

University of Chinese Academy of Sciences(中国科学院大学);Meituan(美团)

Shufan Shen, Zhongni Hou, Chao Bi, Yufei Zhang, Guojun Yin, Shuhui Wang

AI总结:

针对在线策略自蒸馏(OPSD)训练中性能无法持续提升的问题,提出教师进化自蒸馏(TESD),通过面向教师的优化与自适应面向学生的优化提升监督质量,实验显示其性能优于相关代表性方法。

摘要:

在线策略自蒸馏(OPSD)可将学生模型的词元级分布与由特权上下文触发的自教师分布对齐,是缓解序列级后训练范式中奖励稀疏问题的有前景方向。然而,OPSD在训练过程中常无法实现持续性能提升,本文发现模型有限的自教学能力是该现象的根源——其在部分样本上生成不可靠指导。为解决此问题,我们提出教师进化自蒸馏(TESD),使自教师能生成rollout并基于任务正确性与上下文独立性获取环境反馈。首先,将这些奖励作为训练目标,执行面向教师的优化以在蒸馏过程中持续提升模型的自教学能力;其次,通过这些奖励估计教师指导的可靠性,执行自适应面向学生的优化以降低不可靠指导样本的权重。在多个数据集与模型规模上的大量实验表明,TESD始终优于代表性的词元级与序列级方法,凸显了提升OPSD中监督质量的必要性。

英文摘要:

On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity in sequence-level post-training paradigms. However, OPSD often fails to achieve continuous performance gains during training. In this paper, we find that the model's limited self-teaching capability underlies this phenomenon by generating unreliable guidance on certain samples. To address this problem, we propose Teacher-Evolution Self-Distillation (TESD), which enables the self-teacher to generate rollouts and receive environmental feedback based on task correctness and context independence. First, using these rewards as training objectives, we perform teacher-oriented optimization to continuously enhance the model's self-teaching capability during distillation. Second, by estimating teacher guidance reliability from these rewards, we apply adaptive student-oriented optimization to down-weight samples with unreliable guidance. Extensive experiments across multiple datasets and model scales show that TESD consistently outperforms representative token-level and sequence-level methods, highlighting the necessity of improving supervision quality in OPSD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑