arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.29680cs.AIcs.LG
收藏

TraceForge:基于自身思维的蒸馏

TraceForge: Distilling with Your Own Thinking

Yan Wang

AI总结:

提出TraceForge方法,通过两阶段训练生成学生模型风格的推理轨迹,在编码基准上提升了HumanEval+和MBPP+的性能。

摘要:

使用大型语言模型(LLM)为另一个模型生成训练数据,是训练模型最通用的方法之一,至今仍是常见做法。一些前沿团队已对自身模型关闭了该路径:返回的结果是答案或简短摘要,却没有其背后的详细推理轨迹。这有点讽刺:这些模型本身是基于蒸馏人类知识构建的,蒸馏开源模型仍完全是常规操作,该限制仅适用于它们自身的输出。近期研究通过跨模型重放加密块来恢复隐藏推理,这种方法虽有效,但违反了提供者的服务条款,无法长期用于产品构建。我们构建了替代方案:将上下文和教师答案作为输入,以学生模型自身家族的风格生成推理轨迹。训练分为两个阶段:监督阶段和对抗阶段,生成器学习从上下文和答案中再生该家族自身的推理轨迹,判别器学习判断推理轨迹的真实程度。将生成器应用于教师答案时,它会生成学生家族风格的推理轨迹。独立评判者对轨迹的真实可能性按0到1评分:我们生成的轨迹平均得0.68,真实轨迹得0.66,判别器从四个候选中选出的轨迹得0.82,这些是真实性分数而非正确率。我们在五个基准上测试了该思路,其余条件均固定:通用任务上五个训练语料表现相近;编码任务中,生成的推理优于Fable 5自身的思维内容。在相同的5000个问题和相同教师答案下,最终检查点处HumanEval+提升3.7个百分点,MBPP+提升7.1个百分点,且MBPP+的优势在每个检查点均保持。我们推测,学生家族风格的推理更便于其学习。

英文摘要:

Using an LLM to generate training data for another is among the most general ways to train one, and it remains common practice. Some frontier teams have closed that route for their own models: what comes back is an answer, or a short summary, without the detailed reasoning trace behind it. That is a little ironic: they were themselves built on distilling human knowledge, and distilling an open-source model is still entirely routine; the restriction applies to their own outputs alone. Recent work recovers the hidden reasoning by replaying encrypted blocks across models. It somehow works, but it breaks the providers' terms of service, so it is not something a product can be built on long-term. We built the alternative. We take the context and the teacher's answer as input, and generate the reasoning trace ourselves, in the style of the student's own model family. Training is two-stage, supervised and then adversarial: a generator learns to regenerate the family's own reasoning traces from their context and answer, and a critic learns how real a reasoning trace looks. Applied to a teacher's answer, the generator writes one in the family's style. An independent judge rates how likely traces are to be genuine on a scale from 0 to 1: ours average 0.68, real traces 0.66, and traces selected by the critic from four candidates 0.82. These are authenticity scores, not correctness rates. We test the idea on five benchmarks with everything else held fixed. On general tasks the five training corpora are about the same; on coding, the generated reasoning beats Fable 5's own thinking content. With the same 5,000 questions and the same teacher answers, HumanEval+ improves by 3.7 points and MBPP+ by 7.1 at the final checkpoint, and the MBPP+ lead holds at every checkpoint. We suspect that reasoning written in the student's family style is easier for it to learn from.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑