AI 中文总结
本研究探讨对齐训练、模型有机体与玩具模型间SFT经验的三类迁移，发现跨领域迁移SFT经验可推动各领域发展，建议研究者借鉴域外技术。
AI 中文摘要
对齐训练、模型有机体与玩具模型通常被视为独立的研究领域，但这三个领域的项目都常使用监督微调（SFT）追求相同的核心目标。当项目共享同一目标时，应测试从一个领域获得的经验是否可迁移至其他领域。本研究探讨了三类此类迁移，每类均将在一种SFT场景中总结的经验应用于另一场景：首先，将行为泛化相关经验从对齐训练迁移至玩具模型，正如Teaching Claude Why中所做的，基于行为的原因进行训练，相比仅基于行为示例训练，可使行为泛化效果更佳；其次，将能力保留相关经验从模型有机体迁移至Model-Spec Midtraining对齐场景，在训练中使用非学生模型生成的输出（非模型输出）进行SFT会损害能力，而在训练中混入良性的模型内（及策略内）数据，可在嵌入目标行为的同时，防止大部分此类损害；第三，将鲁棒性相关经验从模型有机体迁移至同一对齐场景，研究发现后续的良性SFT会消除对齐行为但保留能力，表明仅能力保留无法确保对后续训练的鲁棒性。本研究说明在不同研究领域间迁移SFT经验可推动各领域发展，建议更多研究者借鉴自身领域之外的技术。
英文摘要
Alignment training, model organisms, and toy models are usually treated as separate research areas. But projects in all three frequently use supervised fine-tuning (SFT) to pursue the same underlying goals. When projects share a goal, we should test whether lessons learned from one area transfer to the other areas. We study three such transfers, each taking a lesson developed in one SFT setting and testing it in another. First, we port a lesson about behavior generalization from alignment training into toy models. Training on the reason for a behavior, as in Teaching Claude Why, can make the behavior generalize better than training on examples of the behavior alone. Second, we port a lesson about capability preservation from model organisms into the Model-Spec Midtraining alignment setting. SFT on outputs written by a model other than the student (off-model outputs) can damage capabilities when trained on. Mixing in benign on-model (and on-policy) data into our training can prevent most of this damage while still embedding the target behavior. Third, we port a lesson about robustness from model organisms into the same alignment setting. We find that follow-up benign SFT can erase the alignment behavior while preserving capabilities, showing that capability preservation alone does not ensure robustness to subsequent training. Our work illustrates how porting SFT lessons between different research fields can uplift them all, suggesting more researchers should borrow techniques from outside their own areas.
Comments31 pages, 13 figures
