基于价值过滤解码的选择性安全引导
Selective Safety Steering via Value-Filtered Decoding
AI总结:
本文提出一种测试时引导方法,通过价值基的安全标准过滤token,减少不必要的干预,提升不安全响应的安全性,同时在多个数据集上验证了其在安全与帮助性之间的更好平衡。
摘要:
尽管大型语言模型(LLMs)被训练以与人类价值观对齐,其生成仍可能违反安全约束。现有工作通过在解码时修改模型的采样策略来解决此问题,但这些方法往往干预过多,修改本应在基模型下安全的生成。本文提出一种新的测试时引导方法,旨在减少此类不必要的干预,同时提高不安全响应的安全性。我们的方法通过价值基的安全标准过滤token,并为误干预的概率提供明确界限。一个单一的阈值超参数控制此界限,使实践者能够权衡更高的不必要干预率以获得更好的输出安全性。在多个数据集和实验中,我们证明了我们的价值过滤解码方法优于现有基线,实现了在安全、帮助性和与基模型相似性之间的更好平衡。
英文摘要:
While large language models (LLMs) are trained to align with human values, their generations may still violate safety constraints. A growing line of work addresses this problem by modifying the model's sampling policy at decoding time using a safety score. However, existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model. Such unnecessary interventions are undesirable, as they can distort key properties of the base model such as helpfulness, fluency, style, and coherence. We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of responses. Our approach filters tokens using a value-based safety criterion and provides an explicit bound on the probability of false interventions. A single threshold hyperparameter controls this bound, allowing practitioners to trade off higher rates of unnecessary intervention for better output safety. Across multiple datasets and experiments, we show that our value-filtered decoding method outperforms existing baselines, achieving better trade-offs between safety, helpfulness, and similarity to the base model.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。