arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09600cs.AIcs.CR
收藏

Antidote:针对有害微调的大语言模型微调后安全对齐方法

Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning

Georgia Institute of Technology(佐治亚理工学院);Dolby Laboratories(杜比实验室)

Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, Ling Liu

AI总结:

针对现有大语言模型安全防御在特定微调超参数下易失效的问题,提出微调后剪枝有害参数的Antidote方法,可在不依赖微调超参数的情况下降低有害评分并保留下游任务性能。

摘要:

经过安全对齐的大语言模型(LLM)易受有害微调攻击——微调数据集中混入少量有害数据即可破坏LLM的安全对齐机制。虽已有多种防御方案被提出,但我们的评估显示,当选择特定训练超参数时,现有防御方法会失效——微调阶段使用较大学习率或较多训练轮次即可轻易让防御失效。为此,我们提出Antidote,一种微调后阶段的解决方案,它对微调阶段的训练超参数完全不敏感。Antidote基于如下理念:通过移除有害参数,就能让有害模型摆脱有害行为,无论这些有害参数在微调阶段是如何形成的。基于该理念,我们在有害微调后引入一个一次性剪枝阶段,以移除负责生成有害内容的有害权重。尽管该方法极为简单,实验结果表明Antidote能够降低有害评分,同时保持下游任务的准确率。代码可在https://github.com/git-disl/Antidote获取。

英文摘要:

Safety aligned Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- a few harmful data mixed in the fine-tuning dataset can break the LLMs's safety alignment. While several defenses have been proposed, our evaluation shows that existing defenses fail \textit{when some specific training hyper-parameters are chosen} -- a large learning rate or a large number of training epochs in the fine-tuning stage can easily invalidate the defense. To this end, we propose Antidote, a post-fine-tuning stage solution, which remains \textbf{\textit{agnostic to the training hyper-parameters in the fine-tuning stage}}. Antidote relies on the philosophy that by removing the harmful parameters, the harmful model can be recovered from the harmful behaviors, regardless of how those harmful parameters are formed in the fine-tuning stage. With this philosophy, we introduce a one-shot pruning stage after harmful fine-tuning to remove the harmful weights that are responsible for the generation of harmful content. Despite its embarrassing simplicity, empirical results show that Antidote can reduce harmful score while maintaining accuracy on downstream tasks. Code is available at https://github.com/git-disl/Antidote.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑