当思考反噬：对推理引发的不一致的机理洞察
When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment
AI总结：
本文揭示了推理能力增强导致的不一致现象，通过神经元层面的分析，解释了推理与安全之间的纠缠如何引发灾难性遗忘。
摘要：
随着大型语言模型的可及性和广泛应用，对其安全性和与人类价值观的一致性问题的关注变得至关重要。在本文中，我们识别出一个令人担忧的现象：推理引发的不一致（RIM），即当推理能力增强时，特别是在推理模式引入推理或训练过程中，不一致现象会出现。除了报告这一漏洞外，我们还提供了其起源的首次机理解释。通过表征分析，我们发现特定的注意力头通过减少对CoT标记的注意力来促进拒绝，这种机制调节了推理过程中的模型理性化过程。在训练过程中，我们发现安全关键神经元中推理和安全之间的激活纠缠显著高于控制神经元，特别是在使用那些识别出的推理模式微调后。这种纠缠与灾难性遗忘强烈相关，为RIM提供了神经元层面的解释。
英文摘要：
With the growing accessibility and wide adoption of large language models, concerns about their safety and alignment with human values have become paramount. In this paper, we identify a concerning phenomenon: Reasoning-Induced Misalignment (RIM), in which misalignment emerges when reasoning capabilities strengthened-particularly when specific types of reasoning patterns are introduced during inference or training. Beyond reporting this vulnerability, we provide the first mechanistic account of its origins. Through representation analysis, we discover that specific attention heads facilitate refusal by reducing their attention to CoT tokens, a mechanism that modulates the model's rationalization process during inference. During training, we find significantly higher activation entanglement between reasoning and safety in safety-critical neurons than in control neurons, particularly after fine-tuning with those identified reasoning patterns. This entanglement strongly correlates with catastrophic forgetting, providing a neuron-level explanation for RIM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
