Lisa:针对有害微调攻击的大语言模型懒安全对齐方法
Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack
AI总结:
针对大语言模型经有害数据微调易被越狱、现有双状态优化方案收敛不稳定的问题,提出引入近端项约束状态漂移的Lisa懒安全对齐方法,理论上保证收敛,实验中可在维持用户任务准确率的同时显著提升对齐性能。
摘要:
近期研究表明,经过安全对齐的大语言模型(LLMs)可通过在混合有害数据的数据集上微调实现越狱。本文首次在相关研究中证实,可通过在微调阶段分离状态以优化对齐数据集与用户数据集,从而缓解越狱效应。但后续研究显示,当投入到对齐状态的步数过少时,这种简单的双状态优化(Bi-State Optimization, BSO)方案会出现收敛不稳定的问题,导致对齐性能下降。通过统计分析,我们发现向共识方向的“过度漂移”可能是造成该不稳定性的原因。为解决这一问题,我们提出懒安全对齐(Lazy(i) safety alignment, Lisa)方法,该方法引入近端项以约束每个状态的漂移。理论上,近端项的优势得到了收敛分析的支持,我们证明足够大的近端因子是保证Lisa收敛的必要条件。实验层面,我们在四个下游微调任务上的结果表明,带有近端项的Lisa能够显著提升对齐性能,同时保持LLM在用户任务上的准确率。代码可在https://github.com/git-disl/Lisa获取。
英文摘要:
Recent studies show that Large Language Models (LLMs) with safety alignment can be jail-broken by fine-tuning on a dataset mixed with harmful data. First time in the literature, we show that the jail-broken effect can be mitigated by separating states in the finetuning stage to optimize the alignment and user datasets. Unfortunately, our subsequent study shows that this simple Bi-State Optimization (BSO) solution experiences convergence instability when steps invested in its alignment state is too small, leading to downgraded alignment performance. By statistical analysis, we show that the \textit{excess drift} towards consensus could be a probable reason for the instability. To remedy this issue, we propose \textbf{L}azy(\textbf{i}) \textbf{s}afety \textbf{a}lignment (\textbf{Lisa}), which introduces a proximal term to constraint the drift of each state. Theoretically, the benefit of the proximal term is supported by the convergence analysis, wherein we show that a sufficient large proximal factor is necessary to guarantee Lisa's convergence. Empirically, our results on four downstream finetuning tasks show that Lisa with a proximal term can significantly increase alignment performance while maintaining the LLM's accuracy on the user tasks. Code is available at \url{https://github.com/git-disl/Lisa}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。