从早期编码到晚期抑制:解释LLM在字符计数任务上的表现
From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks
AI总结:
研究通过字符计数任务揭示LLM内部计算中的结构干扰问题,发现模型在早期层正确编码信息,但后期层通过负电路抑制了正确输出,导致符号推理失败。
摘要:
大型语言模型(LLM)在基本符号任务如单词中的字符计数上表现不佳,尽管在复杂基准测试中表现优异。尽管这一限制已被注意到,但内部原因仍不明确。我们使用字符计数(例如,“苹果中有多少个p?”)作为最小、受控的探针,以隔离令牌级推理与高层混淆因素。在这一设定下,我们发现现代架构(包括LLaMA、Qwen和Gemma)中普遍存在一致现象:模型在内部计算正确答案,但未能在输出层表达。通过结合探测分类器、激活修补、logit镜头分析和注意力头追踪的机理分析,我们发现字符级信息在早期和中期层表示中被编码。然而,这些信息在后期层被一小部分组件衰减,尤其是最后两层MLP。我们识别这些组件为负电路:子网络在正确信号和更高概率但错误输出之间进行权衡。我们的结果导致两个贡献:首先,我们表明LLM的符号推理失败并非由于表示缺失或规模不足,而是源于模型计算图中的结构干扰。这解释了为何此类错误会持续并可能在扩展和指令微调下恶化。其次,我们提供了证据表明LLM前向传递实现了一种竞争解码形式,其中正确和错误假设共存并动态重新加权,最终输出由抑制和放大共同决定。这些发现对可解释性和鲁棒性具有启示:简单的符号推理暴露了现代LLM的弱点,凸显了需要设计策略以确保信息被正确编码并可靠使用的需求。
英文摘要:
Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks. Although this limitation has been noted, the internal reasons remain unclear. We use character counting (e.g., "How many p's are in apple?") as a minimal, controlled probe that isolates token-level reasoning from higher-level confounds. Using this setting, we uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma: models often compute the correct answer internally yet fail to express it at the output layer. Through mechanistic analysis combining probing classifiers, activation patching, logit lens analysis, and attention head tracing, we show that character-level information is encoded in early and mid-layer representations. However, this information is attenuated by a small set of components in later layers, especially the penultimate and final layer MLP. We identify these components as negative circuits: subnetworks that downweight correct signals in favor of higher-probability but incorrect outputs. Our results lead to two contributions. First, we show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph. This explains why such errors persist and can worsen under scaling and instruction tuning. Second, we provide evidence that LLM forward passes implement a form of competitive decoding, in which correct and incorrect hypotheses coexist and are dynamically reweighted, with final outputs determined by suppression as much as by amplification. These findings carry implications for interpretability and robustness: simple symbolic reasoning exposes weaknesses in modern LLMs, underscoring need for design strategies that ensure information is encoded and reliably used.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。