干草堆中寻找神经元：稀疏探针案例研究
Finding Neurons in a Haystack: Case Studies with Sparse Probing
AI总结：
本文通过训练 k-稀疏线性探针分析 LLM 内部神经元激活，揭示不同层的特征表示规律及模型规模对稀疏性的影响。
摘要：
尽管大型语言模型（LLM）已被迅速采用和部署，但这些模型的内部计算仍不透明且鲜为人知。在本研究中，我们试图理解人类可解释的高级特征如何在 LLM 的内部神经元激活中得到表示。我们在这些内部激活上训练 k-稀疏线性分类器（探针），以预测输入中特征的存在；通过改变 k 的取值，我们研究所学表示的稀疏性，以及这种稀疏性如何随模型规模变化。当 k=1 时，我们定位与特定特征高度相关的单个神经元，并开展若干案例研究以说明 LLM 的一般性质。具体而言，我们表明早期层利用神经元的稀疏组合来表示处于叠加状态的许多特征，中间层似乎具有专门的神经元来表示更高级的上下文特征，而规模增大会使表示稀疏性平均上升，但存在多种类型的扩展动态。总体而言，我们在 7 个不同模型中探测了涵盖 10 个不同类别的 100 多种独特特征，这些模型的参数规模从 7000 万到 69 亿不等。
英文摘要：
Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
