少令牌，大杠杆：在微调期间通过约束安全令牌保持安全对齐
Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
AI总结：
提出PACT框架，通过约束安全相关令牌的置信度来防止微调导致的安全对齐漂移，同时保持下游任务性能。
摘要：
大型语言模型（LLMs）通常需要微调（FT）才能在下游任务上表现良好，但即使训练数据集仅包含良性数据，FT也可能导致安全对齐漂移。先前的研究表明，引入少量有害数据会显著损害LLM的拒绝行为，导致LLM顺从有害请求。现有的防御方法通常依赖于模型范围的干预，例如限制哪些参数更新或注入额外的安全数据，这可能会限制通用性并降低下游任务性能。为了解决这些限制，我们提出了一种名为PACT（通过约束令牌保持安全对齐）的微调框架，该框架稳定了模型在安全令牌上的置信度。我们的方法基于经验观察：安全对齐行为反映在模型的令牌级输出置信度中，并且通常集中在少量安全相关令牌上。在下游微调期间，我们正则化微调模型，使其在每一步响应中与对齐参考模型在安全相关令牌上的置信度匹配，同时允许非安全令牌基本不受约束以实现有效的任务适应。这种有针对性的约束防止了对齐漂移，而无需施加通常以牺牲模型效用为代价的全局限制。我们的代码可在{https://github.com/Glresearch1/PACT}获取。
英文摘要：
Large language models (LLMs) often require fine-tuning (FT) to perform well on downstream tasks, but FT can induce safety-alignment drift even when the training dataset contains only benign data. Prior work shows that introducing a small fraction of harmful data can substantially compromise LLM refusal behavior, causing LLMs to comply with harmful requests. Existing defense methods often rely on model-wide interventions, such as restricting which parameters are updated or injecting additional safety data, which can limit generality and degrade downstream task performance. To address these limitations, we propose a fine-tuning framework called Preserving Safety Alignment via Constrained Tokens (PACT), which stabilizes the model's confidence on safety tokens. Our approach is motivated by the empirical observation that safety-aligned behavior is reflected in the model's token-level output confidence and is often concentrated on a small subset of safety-related tokens. During downstream fine-tuning, we regularize the fine-tuned model to match the aligned reference model's confidence on safety-related tokens at each response step, while leaving non-safety tokens largely unconstrained to allow effective task adaptation. This targeted constraint prevents alignment drift without imposing global restrictions that typically trade off with model utility. Our code is available at {https://github.com/Glresearch1/PACT}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
