arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12201cs.LG
收藏

字典学习改进机制可解释性中无补丁的电路发现:以Othello-GPT为例的案例研究

Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT

OpenMOSS Team(OpenMOSS团队);Fudan University(复旦大学)

Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, Xipeng Qiu

AI总结:

提出一种基于稀疏字典学习的无补丁电路发现框架,通过分解残差流模块的字典特征追踪其对模型行为的贡献,在Othello任务上高效发现细粒度可解释电路。

摘要:

稀疏字典学习一直是机制可解释性中快速发展的技术,用于攻克叠加问题并从模型激活中提取更多人类可理解的特征。基于提取出的更多单义特征,我们提出一个进一步的问题:如何识别连接大量字典特征的电路?我们提出了一种替代激活补丁的电路发现框架。我们的框架较少受到分布外问题的影响,并在渐近复杂度方面证明更高效。我们框架中的基本单元是从所有写入残差流的模块(包括嵌入、注意力输出和MLP输出)中分解出的字典特征。从任何logit、字典特征或注意力分数开始,我们设法向下追踪到所有token的低级字典特征,并计算它们对这些更具可解释性和局部性的模型行为的贡献。我们深入研究了在一个名为Othello的合成任务上训练的小型Transformer,并在其中发现了许多人类可理解的细粒度电路。

英文摘要:

Sparse dictionary learning has been a rapidly growing technique in mechanistic interpretability to attack superposition and extract more human-understandable features from model activations. We ask a further question based on the extracted more monosemantic features: How do we recognize circuits connecting the enormous amount of dictionary features? We propose a circuit discovery framework alternative to activation patching. Our framework suffers less from out-of-distribution and proves to be more efficient in terms of asymptotic complexity. The basic unit in our framework is dictionary features decomposed from all modules writing to the residual stream, including embedding, attention output and MLP output. Starting from any logit, dictionary feature or attention score, we manage to trace down to lower-level dictionary features of all tokens and compute their contribution to these more interpretable and local model behaviors. We dig in a small transformer trained on a synthetic task named Othello and find a number of human-understandable fine-grained circuits inside of it.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑