Adapting general-purpose language models to new skills is currently an expensive process that must be repeated as new instruction datasets targeting new skills are created, or can cause the models to forget older skills. In this work, we investigate the effectiveness of adding new skills to preexisting models by training on the new skills in isolation and later merging with the general model (e.g. using task vectors). In experiments focusing on scientific literature understanding, safety, and coding, we find that the parallel-train-then-merge procedure, which is significantly cheaper than retraining the models on updated data mixtures, is often comparably effective. Our experiments also show that parallel training is especially well-suited for enabling safety features in LMs relative to continued finetuning and retraining, as it dramatically improves model compliance with safe prompts while preserving its ability to refuse dangerous or harmful prompts.
Tübingen AI Center, University of Tübingen(图宾根大学图宾根人工智能中心);University of Cambridge(剑桥大学);Munich Center for ML, Technical University of Munich(慕尼黑工业大学慕尼黑机器学习中心)
When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs
Google(谷歌公司);University of New South Wales(新南威尔士大学);University of Technology Sydney(悉尼科技大学);Zhejiang University(浙江大学);Australian National University(澳大利亚国立大学)
Will it Merge? On The Causes of Model Mergeability
Technion - Israel Institute of Technology(技术ion-以色列理工学院);IBM Research AI(IBM人工智能研究部);MIT(麻省理工学院);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室);Hebrew University of Jerusalem(耶路撒冷希伯来大学);Kempner Institute, Harvard University(哈弗大学凯普纳研究所)
Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Merging
Tsinghua University(清华大学);Xidian University(西安电子科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Zhongguancun Laboratory(中关村实验室);National Financial Cryptography Research Center(国家金融密码研究中心)
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学);WeChat AI, Tencent Inc., China(腾讯公司,中国);The University of Hong Kong(香港大学)