仿真失对齐:大语言模型的安全对齐可能适得其反!
Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
AI总结:
本文提出无需训练的仿真失对齐(ED)攻击,通过对比安全对齐模型与预训练模型的输出token分布逆转安全对齐,使模型危害性翻倍,揭示了开源模型安全对齐的潜在风险。
摘要:
大语言模型(LLMs)经过安全对齐,以确保与人类进行安全对话。然而,本文提出了一种无需训练的攻击方法,仅通过访问LLM的输出token分布,即可逆转安全对齐,将更强的对齐结果转化为更大的潜在危害。具体而言,我们的方法通过对比安全对齐语言模型(如Llama-2-chat)与其预训练版本(如Llama-2)的输出token分布来实现这种逆转,从而使token预测向安全对齐的相反方向偏移。我们将此方法命名为仿真失对齐(ED),因为从这种对比分布中采样可证明地模拟了通过微调以最小化安全奖励的结果。我们在三个评估数据集和四个模型家族(Llama-1、Llama-2、Mistral和Alpaca)上使用ED进行的实验表明,ED使预训练模型的危害性翻倍,并优于强基线,在48个评估子集中的43个上以大幅优势取得了最高的有害率。最终,鉴于ED依赖于语言模型输出token分布,这尤其会损害开源模型,我们的发现突显了重新评估语言模型开放可访问性的必要性,即使它们已经过安全对齐。代码可在https://github.com/ZHZisZZ/emulated-disalignment获取。
英文摘要:
Large language models (LLMs) undergo safety alignment to ensure safe conversations with humans. However, this paper introduces a training-free attack method capable of reversing safety alignment, converting the outcomes of stronger alignment into greater potential for harm by accessing only LLM output token distributions. Specifically, our method achieves this reversal by contrasting the output token distribution of a safety-aligned language model (e.g., Llama-2-chat) against its pre-trained version (e.g., Llama-2), so that the token predictions are shifted towards the opposite direction of safety alignment. We name this method emulated disalignment (ED) because sampling from this contrastive distribution provably emulates the result of fine-tuning to minimize a safety reward. Our experiments with ED across three evaluation datasets and four model families (Llama-1, Llama-2, Mistral, and Alpaca) show that ED doubles the harmfulness of pre-trained models and outperforms strong baselines, achieving the highest harmful rates in 43 out of 48 evaluation subsets by a large margin. Eventually, given ED's reliance on language model output token distributions, which particularly compromises open-source models, our findings highlight the need to reassess the open accessibility of language models, even if they have been safety-aligned. Code is available at https://github.com/ZHZisZZ/emulated-disalignment.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。