arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.08473cs.LG
收藏

AsFT:在窄安全盆地内锚定大语言模型微调期间的安全性

AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin

National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学)

Shuo Yang, Qihui Zhang, Yuyang Liu, Xiaojun Jia, Kunpeng Ning, Jiayu Yao, Jigang Wang, Hailiang Dai, Yibing Song, Li Yuan

AI总结:

针对微调大语言模型时安全性易受损的问题,提出AsFT方法,通过惩罚与对齐方向正交的更新,将模型约束在窄安全盆地内,在提升任务性能的同时显著降低有害行为。

摘要:

微调大语言模型(LLMs)可提升性能,但引入了关键的安全漏洞:即使极少的有害数据也会严重破坏安全措施。我们观察到,与对齐方向(由对齐(安全)模型与未对齐模型之间的权重差异定义)正交的扰动会迅速损害模型安全性。相反,沿对齐方向的更新则基本保持安全性,揭示了参数空间是一个“窄安全盆地”。为解决此问题,我们提出AsFT(在微调中锚定安全性),通过在微调过程中显式约束更新方向来维持安全性。通过惩罚与对齐方向正交的更新,AsFT有效将模型约束在“窄安全盆地”内,从而保持其固有安全性。在多个数据集和模型上的大量实验表明,AsFT将有害行为降低高达7.60%,任务性能提升3.44%,并在多个任务上持续优于现有方法。

英文摘要:

Fine-tuning large language models (LLMs) improves performance but introduces critical safety vulnerabilities: even minimal harmful data can severely compromise safety measures. We observe that perturbations orthogonal to the alignment direction - defined by weight differences between aligned (safe) and unaligned models - rapidly compromise model safety. In contrast, updates along the alignment direction largely preserve it, revealing the parameter space as a "narrow safety basin". To address this, we propose AsFT (Anchoring Safety in Fine-Tuning) to maintain safety by explicitly constraining update directions during fine-tuning. By penalizing updates orthogonal to the alignment direction, AsFT effectively constrains the model within the "narrow safety basin," thus preserving its inherent safety. Extensive experiments on multiple datasets and models show that AsFT reduces harmful behaviors by up to 7.60%, improves task performance by 3.44%, and consistently outperforms existing methods across multiple tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑