A key goal of current mechanistic interpretability research in NLP is to find linear features (also called "feature vectors") for transformers: directions in activation space corresponding to concepts that are used by a given model in its computation. Present state-of-the-art methods for finding linear features require large amounts of labelled data -- both laborious to acquire and computationally expensive to utilize. In this work, we introduce a novel method, called "observable propagation" (in short: ObProp), for finding linear features used by transformer language models in computing a given task -- using almost no data. Our paradigm centers on the concept of "observables", linear functionals corresponding to given tasks. We then introduce a mathematical theory for the analysis of feature vectors, including a similarity metric between feature vectors called the coupling coefficient which estimates the degree to which one feature's output correlates with another's. We use ObProp to perform extensive qualitative investigations into several tasks, including gendered occupational bias, political party prediction, and programming language detection. Our results suggest that ObProp surpasses traditional approaches for finding feature vectors in the low-data regime, and that ObProp can be used to better understand the mechanisms responsible for bias in large language models.
Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patterns
Layer by Layer: Uncovering Hidden Representations in Language Models
University of Kentucky(肯塔基大学);Mila-Quebec AI Institute(魁北克AI研究所);University of Montreal(蒙特利尔大学);New York University(纽约大学);University of California, Los Angeles(加州大学洛杉矶分校);Meta FAIR
AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers
Fraunhofer Heinrich-Hertz-Institute(弗劳恩霍夫海因茨·赫兹研究所);Technische Universität Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所(BIFOLD))
What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models
Guangdong University of Technology(广东工业大学);Soochow University(苏州大学);University of Copenhagen(哥本哈根大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal
Indian Institute of Information Technology Allahabad, India(阿利哈巴德信息技术学院,印度);National Institute of Electronics(电子研究所);Indian Institute of Technology Patna, India(帕纳印度理工学院,印度);Indian Institute of Information Technology Kalyani, India(卡里尼信息技术学院,印度)
TimeSynth: A Framework for Uncovering Systematic Biases in Time Series Forecasting
Scientific Computing and Imaging Institute, University of Utah(科学计算与成像研究所,犹他大学);Kahlert School of Computing, University of Utah(Kahlert计算学院,犹他大学);Mountain Biometrics Inc.(山景生物医学公司);Department of Psychiatry, University of Utah(精神病学系,犹他大学)
XAI for Transformers: Better Explanations through Conservative Propagation
Tel-Aviv University(特拉维夫大学);Technische Universität Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(BIFOLD——柏林学习与数据基础研究所);Korea University(高丽大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所)
ELLIS Institute Tübingen(图宾根埃利斯研究所);LIONS, EPFL(日内瓦理工学院LIONS实验室);University of Freiburg(弗赖堡大学);Max-Planck-Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Prior Labs(Prior实验室)
SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity
Broad Institute of MIT and Harvard(麻省理工学院与哈佛 broad 研究所);Shanghai Jiao Tong University(上海交通大学);University of Illinois, Chicago(芝加哥大学伊利诺伊分校);Amazon Web Service(亚马逊网络服务)
Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer
Sorbonne Center for Artificial Intelligence(索邦人工智能中心);Sorbonne Université(索邦大学);Laboratoire de Probabilités, Statistique et Modélisation(概率、统计与建模实验室);CNRS(法国国家科学研究中心);Inria Paris(法国国家信息与自动化研究所巴黎分部)
Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration
National University of Singapore(新加坡国立大学);Hanoi University of Science and Technology(河内科学技术大学);National Institute of Advanced Industrial Science and Technology(国家先进工业科学与技术研究院);Washington State University(华盛顿州立大学)
Uncovering Hidden Systematics in Neural Network Models for High Energy Physics
Bonn-Aachen Institute of Technology, University of Bonn, Germany(波恩-亚琛技术学院,波恩大学,德国);Physics Institute III A, RWTH Aachen University, Germany(亚琛工业大学物理研究所III A,德国);Institute of Physics, University of Bonn, Germany(波恩大学物理研究所,德国);Physics Institute III B, RWTH Aachen University, Germany(亚琛工业大学物理研究所III B,德国)
Radio Foundation Models: Pre-training Transformers for 5G-based Indoor Localization
Fraunhofer Institute for Integrated Circuits (IIS)(弗劳恩霍夫集成电路研究所);Faculty of Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg(弗里德里希-亚历山大-埃尔兰根-纽伦堡大学工程学院)
RIS-Assisted 3D Spherical Splatting for Object Composition Visualization using Detection Transformers
Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki(阿尔忒弥斯大学电气与计算机工程系);Department of Computer Science and Engineering, University of Ioannina(伊奥安纳大学计算机科学与工程系);Department of Electrical and Computer Engineering, Technical University of Crete(克里特技术大学电气与计算机工程系);Institute of Computer Science, Foundation for Research and Technology Hellas(希腊基础研究与技术院计算机科学研究所)