arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.01130cs.AI
收藏

迭代微调大多是自洽的

Iterative Finetuning is Mostly Idempotent

University of Chicago(芝加哥大学)

Zephaniah Roe, Jack Sanderson, Dang Nguyen, Julian Huang, Todd Nief, Aryan Shrivastava, Chenhao Tan, Ari Holtzman

AI总结:

研究通过训练一系列模型,每个模型在前代生成的数据上微调,发现traits在SFT和SDF中衰减或保持不变,而DPO中traits放大需持续训练,但重新初始化后消失,表明持续微调可能引发traits放大,限制此阶段可作为防御手段。

摘要:

如果一个模型具有某种行为倾向,如阿谀或对齐问题,并在其自身输出上进行训练,这种倾向会在下一代模型中被放大吗?我们通过训练一系列模型来研究这个问题,每个模型在前代生成的数据上微调,初始模型带有某些个性或信念。我们测试了三种设置:在指令模型上进行监督微调(SFT)、在基础模型上进行合成文档微调(SDF),以及直接偏好优化(DPO)。在SFT和SDF设置中,traits大多衰减或保持不变,因此进一步微调循环无效果。在极少数情况下,当放大发生时,通常以失去连贯性为代价。在DPO设置中,当模型持续用偏好自身输出的方式训练时,traits放大可以可靠发生,但重新初始化后会消失。总体而言,我们的结果表明,放大很可能来自持续的后训练阶段,限制此阶段可能是一种有效的防御手段。对于非强化学习微调,traits放大罕见且对数据量非常敏感,使其显著不太可能偶然发生。最后,放大与连贯性的权衡作为自然的抑制trait放大的机制。

英文摘要:

If a model has some behavioral tendency, such as sycophancy or misalignment, and it is trained on its own outputs, will the tendency be amplified in the next generation of models? We study this question by training a series of models where each model is finetuned on data generated by its predecessor, and the initial model is seeded with some persona or belief. We test three settings: supervised finetuning (SFT) on instruct models, synthetic document finetuning (SDF) on base models, and direct preference optimization (DPO). In the SFT and SDF settings, traits mostly decay or remain constant so that further finetuning cycles do nothing. In rare cases when amplification occurs, it generally comes at the cost of coherence. In the DPO setting, trait amplification can reliably occur when a model is continually trained with a preference for its own outputs, but vanishes when models are reinitialized at each cycle. Overall, our results suggest that amplification most likely comes from continual post-training, and limiting this stage may be an effective defense. For non-RL finetuning, trait amplification is rare and very sensitive to data quantity, making it significantly less likely to occur accidentally. Finally, the amplification-coherence tradeoff serves as a natural deterrent against trait amplification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑