解构词汇空间中的MLP神经元权重
Disentangling MLP Neuron Weights in Vocabulary Space
AI总结:
本文提出ROTATE方法,通过优化神经元权重旋转以最大化词汇空间的峰度,解构MLP神经元权重,揭示词汇通道,提升语言模型可解释性。
摘要:
解释模型权重中的信息仍是机制可解释性中的基本挑战。本文引入ROTATE(旋转优化的词嵌入对齐),一种无需前向传播的数据自由方法,直接在权重空间中解构MLP神经元。我们的方法基于关键统计观察:编码一致、单义概念的神经元在投影到模型词汇空间时表现出高峰度。通过优化神经元权重的旋转以最大化词汇空间的峰度,我们的方法恢复了稀疏、可解释的方向,称为词汇通道。在Llama-3.1-8B-Instruct和Gemma-2-2B-it上的实验表明,ROTATE一致恢复了忠实于神经元行为的词汇通道。消去单个通道会选择性地禁用相应的输入激活或特定概念的促进。此外,聚合通道级描述可获得全面的神经元描述,在与优化激活基线的头对头比较中表现优异。通过提供神经元权重的数据自由分解,ROTATE为解释语言模型提供了可扩展、细粒度的构建块。
英文摘要:
Interpreting the information encoded in language model weights remains a fundamental challenge in mechanistic interpretability. In this work, we introduce ROTATE (Rotation-Optimized Token Alignment in weighT spacE), a data-free method requiring no forward passes that disentangles MLP neurons directly in weight space. Our approach relies on a key statistical observation: neurons that encode coherent, monosemantic concepts exhibit high kurtosis when projected onto the model's vocabulary. By optimizing rotations of neuron weights to maximize their vocabulary-space kurtosis, our method recovers sparse, interpretable directions which we name vocabulary channels. Experiments on Llama-3.1-8B-Instruct and Gemma-2-2B-it demonstrate that ROTATE consistently recovers vocabulary channels that are faithful to the neuron's behavior; ablating individual channels selectively disables corresponding input activations or the promotion of specific concepts. Moreover, aggregating channel-level descriptions yields comprehensive neuron descriptions that outperform optimized activation-based baselines by 2-3x in head-to-head comparisons. By providing a data-free decomposition of neuron weights, ROTATE offers a scalable, fine-grained building block for interpreting language models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。