\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models. However, existing analyses are done in small models far from the state of the art. To address this, we present a case study of circuit analysis in the 70B Chinchilla model, aiming to test the scalability of circuit analysis. In particular, we study multiple-choice question answering, and investigate Chinchilla's capability to identify the correct answer \emph{label} given knowledge of the correct answer \emph{text}. We find that the existing techniques of logit attribution, attention pattern visualization, and activation patching naturally scale to Chinchilla, allowing us to identify and categorize a small set of `output nodes' (attention heads and MLPs).
We further study the `correct letter' category of attention heads aiming to understand the semantics of their features, with mixed results. For normal multiple-choice question answers, we significantly compress the query, key and value subspaces of the head without loss of performance when operating on the answer labels for multiple-choice questions, and we show that the query and key subspaces represent an `Nth item in an enumeration' feature to at least some extent. However, when we attempt to use this explanation to understand the heads' behaviour on a more general distribution including randomized answer labels, we find that it is only a partial explanation, suggesting there is more to learn about the operation of `correct letter' heads on multiple choice question answering.
From Circuit Evidence to Mechanistic Theory: An Inductive Logic Approach
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);Idiap Research Institute(Idiap研究机构);CRUK National Biomarker Centre, University of Manchester(曼彻斯特大学癌症研究联盟国家生物标志物中心)
Circuit Insights: Towards Interpretability Beyond Activations
Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute(人工智能系,弗劳恩霍夫海因里希·赫兹研究所);Department of Electrical Engineering and Computer Science, Technische Universität Berlin(电气工程与计算机科学系,柏林技术大学);BIFOLD - Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);Centre of eXplainable Artificial Intelligence, Technological University Dublin(可解释人工智能中心,都柏林技术大学)
Posterior-Calibrated Causal Circuits in Variational Autoencoders: Why Image-Domain Interpretability Fails on Tabular Data
Indian Institute of Technology, Patna(印度理工学院巴特那分校);Rajarshi School of Management Technology, Varanasi(拉贾尔希管理技术学院瓦拉纳西分校);Jaypee Institute of Information Technology, Noida(杰皮信息技术学院诺伊达分校)
MINAR: Mechanistic Interpretability for Neural Algorithmic Reasoning
Pacific Northwest National Laboratory, Richland, WA(太平洋西北国家实验室);Halıcıoğlu Data Science Institute, University of California, San Diego, San Diego, CA(哈利奇奥格鲁数据科学研究所,加州大学圣地亚哥分校);Department of Computer and Information Science, University of Pennsylvania, Pennsylvaina, PA(计算机与信息科学系,宾夕法尼亚大学);Department of Mathematics, University of Washington, Seattle, WA(数学系,华盛顿大学)
Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models
MaiNLP, Center for Information and Language Processing, LMU Munich(MaiNLP、信息与语言处理中心、慕尼黑大学);Munich Center for Machine Learning (MCML), Munich(慕尼黑机器学习中心);Language Technology Group, University of Oslo(语言技术组、奥斯陆大学)
SafeSeek: Universal Attribution of Safety Circuits in Language Models
University of Science;United Arab Emirates University;Nanyang Technological University;Zayed University;Intelligent Science \& Technology Academy of CASIC
RESCUE: Repairing Language Model Errors to Sparse Circuits via Reinforcement Learning
Beijing University of Posts and Telecommunications(北京邮电大学);University of Hong Kong(香港大学);Nanyang Technological University(南洋理工大学);China Aerospace Science and Industry Corporation(中国航天科工集团)
PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits
Fraunhofer Heinrich Hertz Institute(弗劳恩霍夫海因里希·赫兹研究所);Technical University of Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所(BIFOLD))
本文提出 PURE 方法,通过识别各纯特征对应的相关回路,将多语义神经元分解为多个单语义虚拟神经元,并在 ImageNet 训练的 ResNet 上验证其优于基于神经元激活的方法。