arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.01185cs.CLcs.AIcs.CR
收藏

令牌级数据选择用于安全大语言模型微调

Token-level Data Selection for Safe LLM Fine-tuning

The Hong Kong University of Science(香港科学与技术大学);School of Data Science, Lingnan University(岭南大学数据科学学院)

Yanping Li, Zhening Liu, Zijian Li, Zehong Lin, Jun Zhang

AI总结:

本文提出TOSS框架,通过令牌级数据选择提升LLM微调的安全性,同时在下游任务性能上优于现有样本级防御方法。

摘要:

对大语言模型(LLM)进行定制数据集的微调已成为适应这些模型到特定领域和应用的标准方法。然而,最近的研究表明,此类微调可能导致模型安全性显著下降。现有的防御方法在样本级别上操作,往往在安全性和实用性之间存在不满意的权衡。为了解决这一限制,我们系统地分析了微调过程中安全性下降的令牌级别。基于此,我们提出了令牌级数据选择用于安全LLM微调(TOSS),一种新的框架,通过测量安全退化模型与实用性导向模型之间的损失差异来量化每个令牌的安全风险。这种令牌级别的细粒度使能够准确识别和移除不安全的令牌,从而保留有价值的任务特定信息。此外,我们引入了一种渐进式细化策略,TOSS-Pro,它通过迭代增强安全退化模型识别不安全令牌的能力。广泛的实验表明,我们的方法在微调过程中稳健地保护LLM,同时在下游任务性能上取得显著优势,明显优于现有的样本级防御方法。我们的代码可在https://github.com/Polly-LYP/TOSS上获得。

英文摘要:

Fine-tuning large language models (LLMs) on custom datasets has become a standard approach for adapting these models to specific domains and applications. However, recent studies have shown that such fine-tuning can lead to significant degradation in the model's safety. Existing defense methods operate at the sample level and often suffer from an unsatisfactory trade-off between safety and utility. To address this limitation, we perform a systematic token-level diagnosis of safety degradation during fine-tuning. Based on this, we propose token-level data selection for safe LLM fine-tuning (TOSS), a novel framework that quantifies the safety risk of each token by measuring the loss difference between a safety-degraded model and a utility-oriented model. This token-level granularity enables accurate identification and removal of unsafe tokens, thereby preserving valuable task-specific information. In addition, we introduce a progressive refinement strategy, TOSS-Pro, which iteratively enhances the safety-degraded model's ability to identify unsafe tokens. Extensive experiments demonstrate that our approach robustly safeguards LLMs during fine-tuning while achieving superior downstream task performance, significantly outperforming existing sample-level defense methods. Our code is available at https://github.com/Polly-LYP/TOSS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑