从组合神经元组中构建可解释特征
Constructing Interpretable Features from Compositional Neuron Groups
AI总结：
本文提出SNMF方法，通过半非负矩阵分解直接分解MLP激活，生成稀疏线性组合的可解释特征，并在多个模型上验证其在因果操控中的优越性。
摘要：
机制可解释性的一个核心目标是识别大型语言模型（LLMs）中能够因果解释其输出的正确分析单位。尽管早期工作聚焦于单个神经元，证据表明神经元常编码多个概念，推动了分析激活空间方向的研究。关键问题是如何在无监督方式下找到捕捉可解释特征的方向。当前方法依赖于字典学习与稀疏自编码器（SAEs），通常在残差流激活上训练以学习方向。然而，SAEs在因果评估中表现不佳且缺乏内在可解释性，因其学习不直接关联模型计算。本文通过直接分解MLP激活，使用半非负矩阵分解（SNMF），使学习特征（a）是激活神经元的稀疏线性组合，（b）映射到其激活输入，从而直接可解释。在Llama 3.1、Gemma 2和GPT-2上的实验表明，SNMF衍生特征在因果操控上优于SAEs和强监督基线（差异均值），并与人类可解释概念一致。进一步分析显示，特定神经元组合在语义相关特征中被重用，揭示了MLP激活空间的层次结构。这些结果使SNMF成为识别可解释特征和分解概念表示的有效工具。
英文摘要：
A central goal for mechanistic interpretability has been to identify the right units of analysis in large language models (LLMs) that causally explain their outputs. While early work focused on individual neurons, evidence that neurons often encode multiple concepts has motivated a shift toward analyzing directions in activation space. A key question is how to find directions that capture interpretable features in an unsupervised manner. Current methods rely on dictionary learning with sparse autoencoders (SAEs), commonly trained over residual stream activations to learn directions from scratch. However, SAEs often struggle in causal evaluations and lack intrinsic interpretability, as their learning is not explicitly tied to the computations of the model. Here, we tackle these limitations by directly decomposing MLP activations with semi-nonnegative matrix factorization (SNMF), such that the learned features are (a) sparse linear combinations of co-activated neurons, and (b) mapped to their activating inputs, making them directly interpretable. Experiments on Llama 3.1, Gemma 2 and GPT-2 show that SNMF derived features outperform SAEs and a strong supervised baseline (difference-in-means) on causal steering, while aligning with human-interpretable concepts. Further analysis reveals that specific neuron combinations are reused across semantically-related features, exposing a hierarchical structure in the MLP's activation space. Together, these results position SNMF as a simple and effective tool for identifying interpretable features and dissecting concept representations in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
