通过分层注意力和激活分析解构Transformer模型中的回忆与推理
Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysis
AI总结：
研究通过可控数据集和激活分析，揭示Transformer模型中回忆与推理的分离电路，为模型泛化和安全部署提供依据。
摘要：
基于语言模型在回忆和推理上的卓越表现，但其内部机制是否独立尚不明确。本文通过机制可解释性方法，利用合成语言谜题数据集，分析Transformer模型在层、头和神经元层面的激活情况。结合激活修补和结构化消融，测量各组件对任务类型的贡献。在两个模型家族（Qwen和LLaMA）中发现，针对不同层和注意力头的干预导致选择性损伤：禁用识别出的'回忆电路'使事实检索准确率下降15%，而推理保持不变；禁用'推理电路'则减少多步推理性能。在神经元层面，观察到任务特定的放电模式，但效果较不稳健，与神经元多义性一致。研究首次提供了因果证据，证明回忆和推理在Transformer模型中依赖于分离但相互作用的电路。这些发现推动了机制可解释性，将电路级结构与功能专业化联系起来，并展示了受控数据集和因果干预如何提供对模型认知的机制见解，从而指导大语言模型的安全部署。
英文摘要：
Transformer-based language models excel at both recall (retrieving memorized facts) and reasoning (performing multi-step inference), but whether these abilities rely on distinct internal mechanisms remains unclear. Distinguishing recall from reasoning is crucial for predicting model generalization, designing targeted evaluations, and building safer interventions that affect one ability without disrupting the other.We approach this question through mechanistic interpretability, using controlled datasets of synthetic linguistic puzzles to probe transformer models at the layer, head, and neuron level. Our pipeline combines activation patching and structured ablations to causally measure component contributions to each task type. Across two model families (Qwen and LLaMA), we find that interventions on distinct layers and attention heads lead to selective impairments: disabling identified "recall circuits" reduces fact-retrieval accuracy by up to 15\% while leaving reasoning intact, whereas disabling "reasoning circuits" reduces multi-step inference by a comparable margin. At the neuron level, we observe task-specific firing patterns, though these effects are less robust, consistent with neuronal polysemanticity.Our results provide the first causal evidence that recall and reasoning rely on separable but interacting circuits in transformer models. These findings advance mechanistic interpretability by linking circuit-level structure to functional specialization and demonstrate how controlled datasets and causal interventions can yield mechanistic insights into model cognition, informing safer deployment of large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
