理解安全对齐：来自安全神经元的机制视角
Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
AI总结：
本文通过机制可解释性识别LLMs中的安全神经元，提出激活对比与修补方法，仅修补5%神经元即可恢复90%以上安全性能，并解释对齐税现象。
摘要：
大型语言模型（LLMs）在各种能力上表现出色，但即使在安全对齐之后，仍会带来生成有害内容和错误信息等安全风险。在本文中，我们通过机制可解释性的视角探索安全对齐的内部机制，重点在于识别和分析LLMs中负责安全行为的安全神经元。我们提出推理时激活对比来定位这些神经元，以及动态激活修补来评估它们对模型安全的因果影响。在多个主流LLMs上的实验表明，我们能够一致地识别出约5%的安全神经元，并且仅通过修补它们的激活，就能在各种红队基准上恢复超过90%的安全性能，而不影响通用能力。安全神经元的发现也有助于解释“对齐税”现象，揭示模型安全和有用性的关键神经元显著重叠，然而这些相同神经元需要不同的激活模式。此外，我们展示了我们的发现在检测不安全输出以保障LLMs安全中的应用。源代码可在https://github.com/THU-KEG/SafetyNeuron获取。
英文摘要：
Large language models (LLMs) excel in various capabilities but pose safety risks such as generating harmful content and misinformation, even after safety alignment. In this paper, we explore the inner mechanisms of safety alignment through the lens of mechanistic interpretability, focusing on identifying and analyzing safety neurons within LLMs that are responsible for safety behaviors. We propose inference-time activation contrasting to locate these neurons and dynamic activation patching to evaluate their causal effects on model safety. Experiments on multiple prevalent LLMs demonstrate that we can consistently identify about $5\%$ safety neurons, and by only patching their activations we can restore over $90\%$ of the safety performance across various red-teaming benchmarks without influencing general ability. The finding of safety neurons also helps explain the ''alignment tax'' phenomenon by revealing that the key neurons for model safety and helpfulness significantly overlap, yet they require different activation patterns for the same neurons. Furthermore, we demonstrate an application of our findings in safeguarding LLMs by detecting unsafe outputs before generation. The source code is available at https://github.com/THU-KEG/SafetyNeuron.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
