Disentangling model activations into meaningful features is a central problem in interpretability. However, the absence of ground-truth for these features in realistic scenarios makes validating recent approaches, such as sparse dictionary learning, elusive. To address this challenge, we propose a framework for evaluating feature dictionaries in the context of specific tasks, by comparing them against \emph{supervised} feature dictionaries. First, we demonstrate that supervised dictionaries achieve excellent approximation, control, and interpretability of model computations on the task. Second, we use the supervised dictionaries to develop and contextualize evaluations of unsupervised dictionaries along the same three axes.
We apply this framework to the indirect object identification (IOI) task using GPT-2 Small, with sparse autoencoders (SAEs) trained on either the IOI or OpenWebText datasets. We find that these SAEs capture interpretable features for the IOI task, but they are less successful than supervised features in controlling the model. Finally, we observe two qualitative phenomena in SAE training: feature occlusion (where a causally relevant concept is robustly overshadowed by even slightly higher-magnitude ones in the learned features), and feature over-splitting (where binary features split into many smaller, less interpretable features). We hope that our framework will provide a useful step towards more objective and grounded evaluations of sparse dictionary learning methods.
A Multi-Level Causal Intervention Framework for Mechanistic Interpretability in Variational Autoencoders
Indian Institute of Technology, Patna（印度理工学院巴特那分校）；Rajarshi School of Management Technology, Varanasi（拉贾尔希管理技术学院瓦拉纳西分校）；Jaypee Institute of Information Technology, Noida（杰皮信息技术学院诺伊达分校）
Amirkabir University of Technology（阿米尔卡比尔理工大学）；Persian Gulf University（波斯湾大学）
Nowadays, deep neural networks are being used in many domains because of their high accuracy results. However, they are considered as "black box", means that they are not explainable for humans. On the other hand, in some tasks such as medical, economic, and self-driving cars, users want the model to be interpretable to decide if they can trust these results or not. In this work, we present a modified version of an autoencoder-based approach for local interpretability called ALIME. The ALIME itself is inspired by a famous method called Local Interpretable Model-agnostic Explanations (LIME). LI…
Universitee de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；LORIA（洛林计算机科学及其应用实验室）
Recent studies have explored the use of deep generative models of speech spectra based of variational autoencoders (VAEs), combined with unsupervised noise models, to perform speech enhancement. These studies developed iterative algorithms involving either Gibbs sampling or gradient descent at each step, making them computationally expensive. This paper proposes a variational inference method to iteratively estimate the power spectrogram of the clean speech. Our main contribution is the analytical derivation of the variational steps in which the en-coder of the pre-learned VAE can be used to e…
Google（谷歌公司）；Universiteit van Amsterdam（阿姆斯特丹大学）；Qualcomm（高通公司）
Variational autoencoders provide a principled framework for learning deep latent-variable models and corresponding inference models. In this work, we provide an introduction to variational autoencoders and some important extensions.
Disentangled Latent Spaces for Reduced Order Models using Deterministic Autoencoders
Institute for Fluid Dynamics and Ship Theory, Hamburg University of Technology（汉堡工业大学流体动力学与船舶理论研究所）；Institute of Mathematics, Hamburg University of Technology（汉堡工业大学数学研究所）
Posterior-Calibrated Causal Circuits in Variational Autoencoders: Why Image-Domain Interpretability Fails on Tabular Data
Indian Institute of Technology, Patna（印度理工学院巴特那分校）；Rajarshi School of Management Technology, Varanasi（拉贾尔希管理技术学院瓦拉纳西分校）；Jaypee Institute of Information Technology, Noida（杰皮信息技术学院诺伊达分校）
Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
Harvard School of Engineering and Applied Science（哈佛工程与应用科学学院）；Kempner Institute for the Study of Natural & Artificial Intelligence（自然与人工智能研究学院）；Harvard Business School（哈佛商学院）
Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）；Delft University of Technology（代尔夫特理工大学）
The dynamics of a turbulent flow tend to occupy only a portion of the phase space at a statistically stationary regime. From a dynamical systems point of view, this portion is the attractor. The knowledge of the turbulent attractor is useful for two purposes, at least: (i) We can gain physical insight into turbulence (what is the shape and geometry of the attractor?), and (ii) it provides the minimal number of degrees of freedom to accurately describe the turbulent dynamics. Autoencoders enable the computation of an optimal latent space, which is a low-order representation of the dynamics. If …
Machine learning and especially deep learning have garneredtremendous popularity in recent years due to their increased performanceover other methods. The availability of large amount of data has aidedin the progress of deep learning. Nevertheless, deep learning models areopaque and often seen as black boxes. Thus, there is an inherent need tomake the models interpretable, especially so in the medical domain. Inthis work, we propose a locally interpretable method, which is inspiredby one of the recent tools that has gained a lot of interest, called localinterpretable model-agnostic explanation…
Disentangling Dense Embeddings with Sparse Autoencoders
The Australian National University（澳大利亚国立大学）；Stanford University（斯坦福大学）；Columbia University（哥伦比亚大学）；Space Telescope Science Institute（空间望远镜科学研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）
University of Gothenburg（哥德堡大学）；Chalmers University of Technology（查尔姆斯理工大学）
High-dimensional data sets are often analyzed and explored via the construction of a latent low-dimensional space which enables convenient visualization and efficient predictive modeling or clustering. For complex data structures, linear dimensionality reduction techniques like PCA may not be sufficiently flexible to enable low-dimensional representation. Non-linear dimension reduction techniques, like kernel PCA and autoencoders, suffer from loss of interpretability since each latent variable is dependent of all input dimensions. To address this limitation, we here present path lasso penalize…
Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders
School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（数据科学学院，香港中文大学（深圳））
Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders
BrainCapture；DTU Health Tech（技术大学丹麦健康技术）；DTU Compute（技术大学丹麦计算）；Department of Biomedical Data Science（生物医学数据科学系）；Department of Computer Science（计算机科学系）；Seer Medical（Seer医疗）；Filadelfia Epilepsy Hospital（菲拉德尔菲亚癫痫医院）；University Hospital of Copenhagen（哥本哈根大学医院）
Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室，智能科学与技术学院，北京大学）
SplInterp: Improving our Understanding and Training of Sparse Autoencoders
School of Mathematics University of Birmingham（数学学院英国伯明翰大学）；Ideami Studios（Ideami工作室）；Department of Mathematics and Statistics University of Limerick（数学与统计学学院英国利默里克大学）；XRAI Inc.（XRAI公司）；Department of Computer Science Brown University（计算机科学系布朗大学）
