arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.04185cs.LGcs.CL
收藏

多层稀疏自编码器的残差流分析

Residual Stream Analysis with Multi-Layer SAEs

Tim Lawson, Lucy Farnik, Conor Houghton, Laurence Aitchison

AI总结:

本研究提出多层稀疏自编码器(MLSAE)以分析Transformer残差流跨层信息流动,发现潜在变量激活层分布方差大且随模型增大而增强,为理解表示变化提供新方法。

摘要:

稀疏自编码器(SAEs)是解释Transformer语言模型内部表示的一种有前景的方法。然而,SAEs通常在每个Transformer层上分别训练,这使得难以利用它们来研究信息如何在层间流动。为了解决这个问题,我们引入了多层稀疏自编码器(MLSAE):一个在来自每个Transformer层的残差流激活向量上训练的单一SAE。鉴于残差流被理解为在层间保留信息,我们预期MLSAE的潜在变量会在某个token位置“开启”并在后续层中保持激活。有趣的是,我们发现单个潜在变量通常对于给定的token或提示在单个层上激活,但单个潜在变量激活的层可能因不同的token或提示而异。我们通过定义一个层上的分布并考虑其方差来量化这些现象。我们发现,当跨token聚合时,潜在变量激活在层上的分布方差比单个token大约两个数量级。对于更大的底层模型,潜在变量在多个层上激活的程度增加,这与残差流激活向量在相邻层变得更相似的事实一致。最后,我们通过应用预训练的调谐透镜变换来放宽残差流基在每个层上相同的假设,但我们的发现仍然在定性上相似。我们的结果代表了一种理解表示在Transformer中流动时如何变化的新方法。我们在https://github.com/tim-lawson/mlsae上发布了我们用于训练和分析MLSAE的代码。

英文摘要:

Sparse autoencoders (SAEs) are a promising approach to interpreting the internal representations of transformer language models. However, SAEs are usually trained separately on each transformer layer, making it difficult to use them to study how information flows across layers. To solve this problem, we introduce the multi-layer SAE (MLSAE): a single SAE trained on the residual stream activation vectors from every transformer layer. Given that the residual stream is understood to preserve information across layers, we expected MLSAE latents to 'switch on' at a token position and remain active at later layers. Interestingly, we find that individual latents are often active at a single layer for a given token or prompt, but the layer at which an individual latent is active may differ for different tokens or prompts. We quantify these phenomena by defining a distribution over layers and considering its variance. We find that the variance of the distributions of latent activations over layers is about two orders of magnitude greater when aggregating over tokens compared with a single token. For larger underlying models, the degree to which latents are active at multiple layers increases, which is consistent with the fact that the residual stream activation vectors at adjacent layers become more similar. Finally, we relax the assumption that the residual stream basis is the same at every layer by applying pre-trained tuned-lens transformations, but our findings remain qualitatively similar. Our results represent a new approach to understanding how representations change as they flow through transformers. We release our code to train and analyze MLSAEs at https://github.com/tim-lawson/mlsae.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑