arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.09055cs.CL
收藏

一种面向子空间的模型融合式大语言模型安全重对齐框架

A safety realignment framework via subspace-oriented model fusion for large language models

School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院)

Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, Liang He

AI总结:

针对大语言模型安全防护易被破解、下游微调后安全能力下降且安全微调易引发灾难性遗忘的问题,提出基于子空间导向模型融合的安全重对齐框架,在保留安全性的同时不显著损害下游任务性能。

摘要:

当前大语言模型(LLMs)的安全防护机制确实容易遭受越狱攻击,存在固有脆弱性。即便是在看似无害的数据上为下游任务进行微调的过程,也可能损害安全性。一种潜在解决方案是在下游微调之后开展安全微调,但安全微调过程中存在灾难性遗忘的风险:LLMs可能恢复了安全防护能力,却丢失了在下游微调中习得的任务特定知识。本文提出一种基于子空间导向模型融合(SOMF)的安全重对齐框架,旨在将初始对齐模型的安全防护能力与当前微调模型的能力相结合,得到重对齐模型。我们的方法首先从每个微调模型的权重中解耦出所有任务向量,随后通过子空间掩码技术在这些向量中识别出安全相关区域,最后基于识别出的安全子空间,探索将初始安全对齐的LLM与所有任务向量进行融合。我们验证了该安全重对齐框架在单个微调模型以及多模型融合场景下均能满足安全要求。研究结果证实,SOMF在保留安全性的同时,不会显著损害下游任务性能,涵盖中文、英文、印地语的指令遵循能力,以及代码和数学领域的问题求解能力。

英文摘要:

The current safeguard mechanisms for large language models (LLMs) are indeed susceptible to jailbreak attacks, making them inherently fragile. Even the process of fine-tuning on apparently benign data for downstream tasks can jeopardize safety. One potential solution is to conduct safety fine-tuning subsequent to downstream fine-tuning. However, there's a risk of catastrophic forgetting during safety fine-tuning, where LLMs may regain safety measures but lose the task-specific knowledge acquired during downstream fine-tuning. In this paper, we introduce a safety realignment framework through subspace-oriented model fusion (SOMF), aiming to combine the safeguard capabilities of initially aligned model and the current fine-tuned model into a realigned model. Our approach begins by disentangling all task vectors from the weights of each fine-tuned model. We then identify safety-related regions within these vectors by subspace masking techniques. Finally, we explore the fusion of the initial safely aligned LLM with all task vectors based on the identified safety subspace. We validate that our safety realignment framework satisfies the safety requirements of a single fine-tuned model as well as multiple models during their fusion. Our findings confirm that SOMF preserves safety without notably compromising performance on downstream tasks, including instruction following in Chinese, English, and Hindi, as well as problem-solving capabilities in Code and Math.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑