arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.12937cs.CLcs.LG
收藏

合并以学习:利用模型合并高效地为语言模型添加技能

Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging

Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)

Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Pang Wei Koh, Jesse Dodge, Pradeep Dasigi

AI总结:

本文提出并行训练新技能后与通用模型合并的方法,以更低成本为语言模型添加科学文献理解、安全性和编程等技能,并证明该方法在保持拒绝有害提示能力的同时显著提升安全遵从性。

摘要:

使通用语言模型适应新技能目前是一个昂贵的过程,每当针对新技能创建新的指令数据集时都必须重复这一过程,否则可能导致模型遗忘旧技能。在本工作中,我们研究了通过孤立地在新技能上训练、随后与通用模型合并(例如使用任务向量)来为已有模型添加新技能的有效性。在聚焦于科学文献理解、安全性和编程的实验中,我们发现并行训练后合并的流程通常与在更新后的数据混合上重新训练模型效果相当,而成本显著更低。我们的实验还表明,相对于持续微调和重新训练,并行训练尤其适合在语言模型中启用安全特性,因为它能显著提高模型对安全提示的遵从性,同时保持其拒绝危险或有害提示的能力。

英文摘要:

Adapting general-purpose language models to new skills is currently an expensive process that must be repeated as new instruction datasets targeting new skills are created, or can cause the models to forget older skills. In this work, we investigate the effectiveness of adding new skills to preexisting models by training on the new skills in isolation and later merging with the general model (e.g. using task vectors). In experiments focusing on scientific literature understanding, safety, and coding, we find that the parallel-train-then-merge procedure, which is significantly cheaper than retraining the models on updated data mixtures, is often comparably effective. Our experiments also show that parallel training is especially well-suited for enabling safety features in LMs relative to continued finetuning and retraining, as it dramatically improves model compliance with safe prompts while preserving its ability to refuse dangerous or harmful prompts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑