arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.09760cs.LG
收藏

针对性疫苗:通过逐层扰动实现大语言模型抵御有害微调的安全对齐

Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation

South China University of Technology(华南理工大学);Pengcheng Laboratory(鹏城实验室);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)

Guozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo, Qi Mu, Li Shen

AI总结:

针对有害微调攻击,提出T-Vaccine安全对齐方法,利用梯度范数识别安全关键层并仅对其施加扰动,在提升防御有效性和资源效率的同时,首次实现消费级GPU上7B模型的防御。

摘要:

有害微调攻击对在线微调服务构成了严重威胁。Vaccine是一种近期的安全对齐阶段防御方法,它对嵌入的所有层施加均匀扰动,使模型对模拟的嵌入漂移具有鲁棒性。然而,应用逐层均匀扰动可能会对某些特定的安全无关层造成过度扰动,导致防御性能下降和不必要的内存消耗。为了解决这一局限性,我们提出了Targeted Vaccine(T-Vaccine),这是一种内存高效的安全对齐方法,仅对模型的选定层施加扰动。T-Vaccine遵循两个核心步骤:首先,它使用梯度范数作为统计度量来识别安全关键层。其次,T-Vaccine不再对所有层施加均匀扰动,而是仅对安全关键层施加扰动,同时在训练期间保持其他层冻结。结果表明,T-Vaccine在防御有效性和资源效率方面均优于Vaccine。与其他防御基线(例如RepNoise和TAR)的比较也证明了T-Vaccine的优越性。值得注意的是,T-Vaccine是首个能够在内存受限的消费级GPU(例如RTX 4090)上,解决7B预训练模型有害微调问题的防御方法。我们的代码可在https://github.com/Llslland/T-Vaccine获取。

英文摘要:

Harmful fine-tuning attack poses a serious threat to the online fine-tuning service. Vaccine, a recent alignment-stage defense, applies uniform perturbation to all layers of embedding to make the model robust to the simulated embedding drift. However, applying layer-wise uniform perturbation may lead to excess perturbations for some particular safety-irrelevant layers, resulting in defense performance degradation and unnecessary memory consumption. To address this limitation, we propose Targeted Vaccine (T-Vaccine), a memory-efficient safety alignment method that applies perturbation to only selected layers of the model. T-Vaccine follows two core steps: First, it uses gradient norm as a statistical metric to identify the safety-critical layers. Second, instead of applying uniform perturbation across all layers, T-Vaccine only applies perturbation to the safety-critical layers while keeping other layers frozen during training. Results show that T-Vaccine outperforms Vaccine in terms of both defense effectiveness and resource efficiency. Comparison with other defense baselines, e.g., RepNoise and TAR also demonstrate the superiority of T-Vaccine. Notably, T-Vaccine is the first defense that can address harmful fine-tuning issues for a 7B pre-trained models trained on consumer GPUs with limited memory (e.g., RTX 4090). Our code is available at https://github.com/Lslland/T-Vaccine.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑