arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.04524cs.CL
收藏

迈向安全调优:缓解指令微调带来的安全风险

Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuning

Harbin Institute of Technology(哈尔滨工业大学);The Chinese University of Hong Kong(香港中文大学)

Yanrui Du, Sendong Zhao, Jiawei Cao, Ming Ma, Danyang Zhao, Shuren Qi, Fenglei Fan, Ting Liu, Bing Qin

AI总结:

针对指令微调引入的 LLM 安全风险,本文提出 SWAT 训练中防御策略,通过优先预热稳健模块 Mods_Rob 再全参数训练,在保持任务性能的同时降低安全特征漂移,并可兼容现有训练前后防御。

摘要:

指令微调已成为为特定应用定制大型语言模型(LLM)的一项关键技术。然而,近期研究强调了经过微调的 LLM 存在显著安全漏洞。现有防御工作更多聚焦于训练前和训练后方法,而训练中方法仍未得到充分探索。为填补这一空白,我们提出了一种名为 SWAT 的新型安全调优策略。通过分析模块级参数(例如 Q/K/V/O)如何影响安全特征空间漂移,我们识别出一个稳健的模块子集,称为 Mods_Rob。我们的 SWAT 策略首先预热 Mods_Rob,以最小的安全风险捕获低层特征,随后训练所有参数以获得最佳任务性能。本质上,该策略将早期学习负担更多地从全局参数转移到 Mods_Rob,从而降低非稳健子集的更新幅度。在多种数据集、场景和 LLM 上,我们的策略已证明能在保持任务性能的同时显著缓解安全风险。重要的是,它可以与训练前和训练后方法无缝集成,从而带来更大提升。

英文摘要:

Instruction fine-tuning has emerged as a critical technique for customizing Large Language Models (LLMs) to specific applications. However, recent studies have highlighted significant security vulnerabilities in fine-tuned LLMs. Existing defense efforts focus more on pre-training and post-training methods, yet there remains underexplored in in-training methods. To fill this gap, we introduce a novel secure-tuning strategy called SWAT. By analyzing how module-level parameters (e.g. Q/K/V/O) affect the security feature space drift, we identify a robust subset of modules, termed Mods_Rob. Our SWAT strategy begins by warming up Mods_Rob to capture low-level features with minimal security risks, followed by training all parameters to achieve optimal task performance. Essentially, this strategy shifts the early learning burden more from global parameters to Mods_Rob, reducing update magnitudes of the non-robust subset. Across various datasets, scenarios, and LLMs, our strategy has demonstrated significant success in mitigating security risks while preserving task performance. Importantly, it can be seamlessly integrated with pre-training and post-training methods, leading to greater improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑