大语言模型如何被说服:几个被重定向的注意力头
How LLMs Are Persuaded: A Few Attention Heads, Rerouted
AI总结:
研究揭示大语言模型在被说服时,少量中间层注意力头决定答案,通过重定向注意力产生事实性错误,该机制在开源LLM和现实攻击场景中均有效。
摘要:
语言模型可以被说服放弃事实性知识。这种漏洞对AI安全至关重要,但其内部机制尚不明确。我们发现了一个紧凑的因果机制,解释说服导致的事实性错误。少量中间层注意力头几乎完全决定了模型的答案。这些头将答案选项写入一个低维多面体,选项占据不同的顶点。说服不模糊信念或仅仅降低置信度;它导致从正确答案顶点到说服目标顶点的离散潜在跳跃。我们展示决策头并非基于证据推理,而是复制其注意力选择的选项标记。说服通过重定向注意力实现。我们隔离了一个秩一的证据路由特征,控制路由路径。直接修改此特征可引导模型选择,移除则阻止说服。随后我们追溯此特征到更浅层的注意力头,这些头从输入中的说服关键词构建它。每一步均通过干预验证。该机制在开源LLM和现实攻击场景如生成引擎优化中均出现,揭示说服是一种狭窄、可监控的电路。
英文摘要:
Large language models can answer a factual question correctly, yet switch to an incorrect answer when exposed to persuasive context. Why this happens inside the model remains unclear. In this work, we study this question through causal interventions in controlled multiple-choice tasks and identify that persuasion mainly acts through a small set of mid-layer attention heads, which we call decision heads. These heads do not compute the answer themselves. Instead, they copy the option they attend to, so changing their attention can directly change the model's answer. We find that this attention is controlled by a one-dimensional option-routing feature in the option-token representations. Persuasive text changes this feature, causing the decision heads to attend to the persuasion target instead of the correct option. We further trace the routing feature to shallower attention heads that read persuasive keywords from the input. Together, these results reveal a causal pathway: shallow heads read persuasive keywords and write the routing feature, which redirects the decision heads' attention toward the persuasion target; the decision heads then copy the selected option, leading to the wrong answer. We verify each step through intervention and find the same mechanism across four open-source model families and in a source-selection task derived from Generative Engine Optimization (GEO). Finally, based on this mechanism, we introduce a low-rank update to the decision heads' key projections to reduce their sensitivity to persuasion.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。