深度对齐但不确定性依然存在：通过将不安全行为归因于基础模型来改进推理时间的安全性
Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model
AI总结：
本文研究了对齐对语言模型安全性和通用性的影响，提出BoN采样方法将不安全行为归因于基础模型，减少多个安全基准的攻击成功率，同时保持实用性。
摘要：
尽管拒绝训练在大语言模型（LLMs）中的广泛应用展示了模型安全性的提升，但最近的研究指出现有对齐方法的浅层性质存在不足。为此，本文提出的深度对齐方法通过从更强的推理模型中蒸馏推理能力，从而在LLMs中植入更深层次的安全性。本文研究了深度对齐对语言模型的影响。首先，我们表明尽管教师模型在规模和安全性方面更大更强，但教师和学生模型之间仍存在对齐差距，这会影响学生模型的安全性和通用性。此外，我们表明通过深度对齐对齐的模型仍可能保留来自基础模型的不安全行为，尽管它们学习了更大推理模型的推理模式。基于这一观察，我们提出了一种BoN采样方法，该方法在潜在空间中将不安全行为归因于基础LLM，从而降低不安全响应的排名，从而在多个安全基准上获得有意义的安全性改进，同时最小化实用性损失。特别是，在7个教师模型和6个不同类别和规模的学生模型上，我们展示了在DAN、WildJailbreak和StrongREJECT基准上平均攻击成功率（ASR）的减少分别为28.2%、31.3%和35.4%。我们进一步表明，这些安全性提升在RL训练后仍然存在，从而突显了安全性推理的不确定性，并明确将其归因于基础模型。
英文摘要：
While the wide adoption of refusal training in large language models (LLMs) has showcased improvements in model safety, recent works have highlighted shortcomings due to the shallow nature of these alignment methods. To this end, the work on Deliberative alignment proposed distilling reasoning capabilities from stronger reasoning models, thereby instilling deeper safety in LLMs. In this work, we study the impact of deliberative alignment in language models. First, we show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models, which affects both the safety and general utility of the student model. Furthermore, we show that models aligned through deliberative alignment can retain unsafe behaviors from the base model despite learning the reasoning patterns of larger reasoning models. Building upon this observation, we propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space, thereby down-ranking unsafe responses to gain a meaningful improvement in model safety across multiple safety benchmarks with minimal loss in utility. In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks. We further show that these safety gains prevail post RL training, thus highlighting the uncertainty in safety reasoning and it's explicit attribution to the base model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
