学习保持安全:在微调过程中对抗安全退化的一种自适应正则化方法
Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning
AI总结:
本文提出一种自适应正则化框架,通过评估安全风险来维持模型在微调过程中的安全性,实验表明该方法能有效降低攻击成功率并保持下游性能。
摘要:
遵循指示的语言模型在训练时被设计为有用且安全,但其安全行为在温和的微调下会退化,在对抗性更新下会恶化。现有的防御方法往往提供有限的保护或迫使安全与实用性之间做出权衡。我们介绍了一种训练框架,该框架能够根据安全风险自适应地调整正则化,使模型在整个微调过程中保持对齐。为了在训练时间内估计安全风险,我们探索了两种不同的方法:基于判断的Safety Critic,它为训练批次分配高层危害评分;以及基于激活的风险预测器,该预测器利用一个轻量级分类器,该分类器在中间模型激活上进行训练以估计有害意图。每种方法提供一个风险信号,用于约束被认为高风险的更新保持接近安全参考策略,而低风险的更新则继续标准训练。我们实证验证了有害意图信号可以从预生成激活中预测,并且判断评分提供了有效的高召回率安全指导。在多个模型家族和攻击场景中,与标准微调相比,使用任一风险估计方法的自适应正则化一致降低了攻击成功率,保持了下游性能,并不增加推理时间成本。这项工作展示了一种维持安全而不牺牲实用性的原理性机制。
英文摘要:
Instruction-following language models are trained to be helpful and safe, yet their safety behavior can deteriorate under benign fine-tuning and worsen under adversarial updates. Existing defenses often offer limited protection or force a trade-off between safety and utility. We introduce a training framework that adapts regularization in response to safety risk, enabling models to remain aligned throughout fine-tuning. To estimate safety risk at training time, we explore two distinct approaches: a judge-based Safety Critic that assigns high-level harm scores to training batches, and an activation-based risk predictor built with a lightweight classifier trained on intermediate model activations to estimate harmful intent. Each approach provides a risk signal that is used to constrain updates deemed higher risk to remain close to a safe reference policy, while lower-risk updates proceed with standard training. We empirically verify that harmful intent signals are predictable from pre-generation activations and that judge scores provide effective high-recall safety guidance. Across multiple model families and attack scenarios, adaptive regularization with either risk estimation approach consistently lowers attack success rate compared to standard fine-tuning, preserves downstream performance, and adds no inference-time cost. This work demonstrates a principled mechanism for maintaining safety without sacrificing utility.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。