A key goal in mechanistic interpretability is circuit analysis: finding sparse subgraphs of models corresponding to specific behaviors or capabilities. However, MLP sublayers make fine-grained circuit analysis on transformer-based language models difficult. In particular, interpretable features -- such as those found by sparse autoencoders (SAEs) -- are typically linear combinations of extremely many neurons, each with its own nonlinearity to account for. Circuit analysis in this setting thus either yields intractably large circuits or fails to disentangle local and global behavior. To address this we explore transcoders, which seek to faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer. We introduce a novel method for using transcoders to perform weights-based circuit analysis through MLP sublayers. The resulting circuits neatly factorize into input-dependent and input-invariant terms. We then successfully train transcoders on language models with 120M, 410M, and 1.4B parameters, and find them to perform at least on par with SAEs in terms of sparsity, faithfulness, and human-interpretability. Finally, we apply transcoders to reverse-engineer unknown circuits in the model, and we obtain novel insights regarding the "greater-than circuit" in GPT2-small. Our results suggest that transcoders can prove effective in decomposing model computations involving MLPs into interpretable circuits. Code is available at https://github.com/jacobdunefsky/transcoder_circuits/.
CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Jinesis AI Lab(Jinesis人工智能实验室);University of Toronto(多伦多大学);Vector Institute(向量研究所);CMU(卡内基梅隆大学);EuroSafeAI;ELLIS Institute Tübingen(图宾根ELLIS研究所)
Circuit Insights: Towards Interpretability Beyond Activations
Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute(人工智能系,弗劳恩霍夫海因里希·赫兹研究所);Department of Electrical Engineering and Computer Science, Technische Universität Berlin(电气工程与计算机科学系,柏林技术大学);BIFOLD - Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);Centre of eXplainable Artificial Intelligence, Technological University Dublin(可解释人工智能中心,都柏林技术大学)
How Do Linear Probes Emerge? A Circuit-Tracing Framework with Concept-Targeted Attribution
University of Toronto(多伦多大学);Vector Institute(矢量研究院);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所)
Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);School of Artificial Intelligence for Science, Peking University(北京大学科学人工智能学院)
Modular Arithmetic: Language Models Solve Math Digit by Digit
German Research Center for AI (DFKI)(德国人工智能研究中心(DFKI));Saarland University(萨尔兰大学);Center for European Research in Trusted AI (CERTAIN)(可信人工智能欧洲研究中心(CERTAIN))
CircuitLM: A Multi-Agent LLM-Aided Design Framework for Generating Circuit Schematics from Natural Language Prompts
Department of Computer Science and Engineering(计算机科学与工程系);Department of Electrical and Electronic Engineering(电气与电子工程系);Islamic University of Technology(伊斯兰技术大学)
Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Video
Mila Quebec(蒙特利尔大学);McGill University(麦吉尔大学);Meta;Université de Montréal(蒙特利尔大学);Imperial College London(伦敦帝国理工学院);Fraunhofer Heinrich Hertz Institute(弗劳恩霍夫 Heinrich Hertz 研究所);Technological University Dublin(都柏林技术大学);Apollo Research(Apollo 研究所)