技能条件门控自蒸馏用于大语言模型推理
Skill-Conditioned Gated Self-Distillation for LLM Reasoning
AI总结：
提出技能条件门控自蒸馏（SGSD），通过从经验技能库中检索技能-错误对构建多教师池，并利用验证器验证教师极性，以鲁棒门控目标蒸馏信息性师生差异，在弱先验信息假设下提升数学推理性能。
摘要：
在线自蒸馏（SD）通过使用教师端特权信息（PI）将稀疏的验证器结果转化为密集的令牌级监督，从而改善大语言模型推理。现有方法通常假设可信的PI，例如参考答案或成功轨迹。我们提出PI是否可以来自经验驱动的技能库，其中检索到的技能紧凑且可重用，但也可能不相关或具有误导性。我们提出技能条件门控自蒸馏（SGSD），将基于技能的SD表述为教师假设验证而非无条件模仿。SGSD检索技能-错误对，构建多教师池，并让所有技能条件教师对相同的普通提示学生输出进行评分。验证器验证每个教师的极性：支持成功或抑制失败提供正向监督，而相反立场则被反转。然后，一个鲁棒的门控目标蒸馏信息性的师生差异，同时抑制不确定或极端信号。在多个数学推理基准上的实验表明，SGSD在弱PI假设下持续优于GRPO，并与答案条件OPSD保持竞争力。例如，在Qwen3-1.7B上，SGSD在AIME24、AIME25和HMMT25上平均比GRPO高出6.2%，比OPSD高出1.7%。我们的代码可在https://github.com/walawalagoose/SGSD获取。
英文摘要：
On-policy self-distillation (SD) improves LLM reasoning by using teacher-side privileged information (PI) to turn sparse verifier outcomes into dense token-level supervision. Existing methods usually assume trusted PI, such as reference answers or successful traces. We ask whether PI can instead come from an experience-derived skill bank, where retrieved skills are compact and reusable but may also be irrelevant or misleading. We propose Skill-Conditioned Gated Self-Distillation (SGSD), which formulates skill-based SD as teacher hypothesis validation rather than unconditional imitation. SGSD retrieves skill-mistake pairs, constructs a multi-teacher pool, and lets all skill-conditioned teachers score the same plain-prompt student rollout. The verifier validates each teacher's polarity: supporting a success or suppressing a failure gives positive supervision, while the opposite stance is reversed. A robust gated objective then distills informative teacher-student disagreements while suppressing uncertain or extreme signals. Experiments on multiple mathematical reasoning benchmarks show that SGSD consistently improves over GRPO and remains competitive with answer-conditioned OPSD under a weaker PI assumption. For example, on Qwen3-1.7B, SGSD outperforms GRPO by 6.2% and OPSD by 1.7% on average on AIME24, AIME25, and HMMT25.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
