手术:通过注意力sink缓解大型语言模型的有害微调
Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink
AI总结:
Surgery通过注意力sink机制在微调阶段抑制有害模式学习,提升模型安全性。
摘要:
有害微调可能会破坏大型语言模型的安全对齐,从而暴露显著的安全风险。在本文中,我们利用注意力sink机制来缓解有害微调。具体来说,我们首先测量每个注意力头的一个统计量,称为sink divergence,并观察到不同的注意力头表现出sink divergence的两种不同符号。为了了解其安全影响,我们进行了实验,发现随着模型在有害微调中变得越来越有害,具有正sink divergence的注意力头数量也随之增加。基于这一发现,我们提出了一个可分离的sink divergence假设——在微调过程中与学习有害模式相关的注意力头可以通过sink divergence的符号进行分离。基于该假设,我们提出了一种在微调阶段的防御方法,称为Surgery。Surgery利用一个sink divergence抑制的正则化器,将注意力头引导到负sink divergence组,从而减少模型学习和放大有害模式的倾向。大量实验表明,Surgery在BeaverTails、HarmBench和SorryBench基准上分别将防御性能提高了5.90%、11.25%和9.55%。源代码可在https://github.com/Lslland/Surgery上获得。
英文摘要:
Harmful fine-tuning can invalidate safety alignment of large language models, exposing significant safety risks. In this paper, we utilize the attention sink mechanism to mitigate harmful fine-tuning. Specifically, we first measure a statistic named \emph{sink divergence} for each attention head and observe that \emph{different attention heads exhibit two different signs of sink divergence}. To understand its safety implications, we conduct experiments and find that the number of attention heads of positive sink divergence increases along with the increase of the model's harmfulness when undergoing harmful fine-tuning. Based on this finding, we propose a separable sink divergence hypothesis -- \emph{attention heads associating with learning harmful patterns during fine-tuning are separable by their sign of sink divergence}. Based on the hypothesis, we propose a fine-tuning-stage defense, dubbed Surgery. Surgery utilizes a regularizer for sink divergence suppression, which steers attention heads toward the negative sink divergence group, thereby reducing the model's tendency to learn and amplify harmful patterns. Extensive experiments demonstrate that Surgery improves defense performance by 5.90\%, 11.25\%, and 9.55\% on the BeaverTails, HarmBench, and SorryBench benchmarks, respectively. Source code is available on https://github.com/Lslland/Surgery.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。