arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.10264cs.LGcs.CL
收藏

什么使安全微调有效或失效?一项机制性研究

What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

Five AI Ltd.(Five AI有限公司);University of Michigan(密歇根大学);Harvard University(哈佛大学);University of Oxford(牛津大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Copenhagen(哥本哈根大学)

Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, Puneet K. Dokania

AI总结:

本研究通过合成数据框架和机制分析,发现安全微调仅最小调整MLP权重将不安全输入映射到零空间,导致对抗输入被误判为安全,并在Llama-2和Llama-3上验证。

摘要:

安全微调有助于使大型语言模型(LLMs)与人类偏好对齐,以实现其安全部署。为了更好地理解通过安全微调使模型变得安全的潜在因素,我们设计了一个合成数据生成框架,通过建模模型被要求执行的任务(例如“设计”)与任务所针对的具体概念(例如“自行车”与“炸弹”)之间的交互,来捕捉不安全输入的关键方面。利用这一框架,我们研究了三种著名的安全微调方法——监督式安全微调、直接偏好优化和遗忘学习——并提供了重要证据,表明这些方法仅最小程度地调整MLP权重,以专门将不安全输入对齐到其权重的零空间中。这导致模型根据其是否认为输入安全而对输入进行聚类。相应地,当提供对抗性输入(例如越狱提示)时,其激活更接近安全样本,导致模型将此类输入视为安全输入进行处理。我们尽可能在真实世界模型上验证了我们的发现,具体包括Llama-2 7B和Llama-3 8B。

英文摘要:

Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. To better understand the underlying factors that make models safe via safety fine-tuning, we design a synthetic data generation framework that captures salient aspects of an unsafe input by modeling the interaction between the task the model is asked to perform (e.g., "design") versus the specific concepts the task is asked to be performed upon (e.g., a "cycle" vs. a "bomb"). Using this, we investigate three well-known safety fine-tuning methods -- supervised safety fine-tuning, direct preference optimization, and unlearning -- and provide significant evidence demonstrating that these methods minimally transform MLP weights to specifically align unsafe inputs into its weights' null space. This yields a clustering of inputs based on whether the model deems them safe or not. Correspondingly, when an adversarial input (e.g., a jailbreak) is provided, its activations are closer to safer samples, leading to the model processing such an input as if it were safe. We validate our findings, wherever possible, on real-world models -- specifically, Llama-2 7B and Llama-3 8B.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑