arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.34005cs.LGcs.AI
收藏

引导教师:改进思维模型的在线策略自蒸馏

Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models

Arizona State University(亚利桑那州立大学)

Tiejin Chen, Xiaoou Liu, Hanyang Chen, Hua Wei

AI总结:

本研究针对思维模型在线策略自蒸馏的参考幻觉与重新考虑抑制问题,提出引导教师 OPSD 方法,通过激活引导优化教师,在 AIME、HMMT 实验中提升准确率并恢复相关指标至基础模型水平。

摘要:

在线策略自蒸馏(OPSD)利用从同一模型初始化的参考条件教师的 token 级监督,在自身推理轨迹上训练学生模型。然而,参考的存在会导致教师行为破坏思维模型的独立推理,产生两种失效模式:一是参考幻觉,即学生引用不可用的参考来为推理步骤辩护;二是重新考虑抑制,即教师在不确定步骤中阻止验证与修正。我们引入引导教师 OPSD(ST-OPSD),通过激活引导共同解决这些失效模式。ST-OPSD 从模型生成的推理轨迹中提取两个方向,在蒸馏目标计算期间将其应用于教师的隐藏状态,抑制参考调用行为,并通过标记族校准鼓励重新考虑。学生从所得教师分布中学习,不受引导。在 AIME 和 HMMT 上的实验表明,ST-OPSD 相较于普通 OPSD 和其他干预措施提高了准确率,同时将重新考虑率和参考幻觉率恢复到基础模型水平。

英文摘要:

On-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model. However, access to the reference can induce teacher behaviors that undermine independent reasoning in thinking models, leading to reference hallucinations, where the student cites an unavailable reference to justify reasoning steps, and reconsideration suppression, where the teacher discourages verification and revision at uncertain steps. We introduce Steered-Teacher OPSD (ST-OPSD) to jointly address these failure modes through activation steering. ST-OPSD extracts two directions from model-generated reasoning trajectories and applies them to the teacher's hidden states during distillation target computation, suppressing reference-invoking behavior and encouraging reconsideration with marker-family calibration. The student learns from the resulting teacher distributions without being steered. Experiments on AIME and HMMT show that ST-OPSD improves accuracy over vanilla OPSD and other interventions while restoring reconsideration and reference-hallucination rates to base-model levels.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑