SkillFactory：用于学习认知行为的自我蒸馏
SkillFactory: Self-Distillation For Learning Cognitive Behaviors
AI总结：
SkillFactory通过监督微调阶段学习认知技能，为强化学习奠定基础，提升模型在复杂任务中的泛化能力与鲁棒性。
摘要：
推理模型利用长链思考过程时，会使用多种认知技能，如验证答案、回溯、尝试其他方法等。先前研究表明，当基础语言模型具备这些技能时，通过强化学习（RL）进一步训练可使其有效利用这些技能。如何让模型学习基础模型未展现的技能？我们的工作SkillFactory是一种方法，通过监督微调（SFT）阶段让模型大致学习这些技能，再进行RL。我们的方法不依赖更强模型的蒸馏，而是利用模型自身生成的样本，重新排列成技能格式的训练数据。这些“银色”SFT轨迹可能不完美，但能有效激发模型在RL中获取技能。评估显示：（1）SkillFactory SFT初始化有助于模型在RL后泛化至更难任务，尽管RL前表现较低；（2）模型确实使用了认知技能；（3）RL后的SkillFactory模型在领域外任务中比RL后的基础模型更稳健。我们的工作表明，在RL前学习的归纳偏置有助于模型学习稳健的认知技能使用。
英文摘要：
Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL;(2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
