The pursuit of interpretable artificial intelligence has led to significant advancements in the development of methods that aim to explain the decision-making processes of complex models, such as deep learning systems. Among these methods, causal abstraction stands out as a theoretical framework that provides a principled approach to understanding and explaining the causal mechanisms underlying model behavior. This survey paper delves into the realm of causal abstraction, examining its theoretical foundations, practical applications, and implications for the field of model interpretability.
Structural causal models (SCMs) are a widespread formalism to deal with causal systems. A recent direction of research has considered the problem of relating formally SCMs at different levels of abstraction, by defining maps between SCMs and imposing a requirement of interventional consistency. This paper offers a review of the solutions proposed so far, focusing on the formal properties of a map between SCMs, and highlighting the different layers (structural, distributional) at which these properties may be enforced. This allows us to distinguish families of abstractions that may or may not b…
Structural analysis methods (e.g., probing and feature attribution) are increasingly important tools for neural network analysis. We propose a new structural analysis method grounded in a formal theory of causal abstraction that provides rich characterizations of model-internal representations and their roles in input/output behavior. In this method, neural representations are aligned with variables in interpretable causal models, and then interchange interventions are used to experimentally verify that the neural representations have the causal properties of their aligned variables. We apply …
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能）
Learning dynamics models accurately is an important goal for Model-Based Reinforcement Learning (MBRL), but most MBRL methods learn a dense dynamics model which is vulnerable to spurious correlations and therefore generalizes poorly to unseen states. In this paper, we introduce Causal Dynamics Learning for Task-Independent State Abstraction (CDL), which first learns a theoretically proved causal dynamics model that removes unnecessary dependencies between state variables and the action, thus generalizing well to unseen states. A state abstraction can then be derived from the learned dynamics, …
McGill University（麦吉尔大学）；Mila（米拉研究所）；Facebook AI Research（脸书人工智能研究院）；University of Oxford（牛津大学）；Deepmind（深度思维）
Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges. In this paper, we consider the problem of learning abstractions that generalize in block MDPs, families of environments with a shared latent state space and dynamics structure over that latent space, but varying observations. We leverage tools from causal inference to propose a method of invariant prediction to learn model-irrelevance state abstractions (MISA) that generalize to novel observations in the multi-environment setting. We prove that for certain…
University of Strathclyde（思克莱德大学）；University of Copenhagen（哥本哈根大学）
Interventional causal models describe several joint distributions over some variables used to describe a system, one for each intervention setting. They provide a formal recipe for how to move between the different joint distributions and make predictions about the variables upon intervening on the system. Yet, it is difficult to formalise how we may change the underlying variables used to describe the system, say moving from fine-grained to coarse-grained variables. Here, we argue that compositionality is a desideratum for such model transformations and the associated errors: When abstracting…
The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis
Boston University（波士顿大学）；University of Mannheim（曼海姆大学）；Northeastern University（东北大学）；Anthropic（Anthropic公司）；Massachusetts Institute of Technology（麻省理工学院）；Stanford University（斯坦福大学）；Technion – Israel Institute of Technology（技术学院–以色列理工学院）
Boston University（波士顿大学）；Pr(AI)2 R Group（Pr(AI)2 R 研究组）；Ai2（艾伦人工智能研究所）；Technion（以色列理工学院）；University of Buenos Aires（布宜诺斯艾利斯大学）；Northeastern University（东北大学）；Brown University（布朗大学）；University of Amsterdam（阿姆斯特丹大学）；MIT（麻省理工学院）；Cambridge University（剑桥大学）
本文提出机制可解释性基准 MIB，通过电路定位与因果变量定位两赛道系统比较方法，发现归因/掩码优化及有监督 DAS 分别表现最佳。
Arizona State University（亚利桑那州立大学）；Army Research Lab（陆军研究实验室）
Machine learning models have had discernible achievements in a myriad of applications. However, most of these models are black-boxes, and it is obscure how the decisions are made by them. This makes the models unreliable and untrustworthy. To provide insights into the decision making processes of these models, a variety of traditional interpretable models have been proposed. Moreover, to generate more human-friendly explanations, recent work on interpretability tries to answer questions related to causality such as "Why does this model makes such decisions?" or "Was it a specific feature that …
Kyoto University（京都大学）；Nagahama Institute of Bio-Science and Technology（长滨生物科学技术大学）
We develop a category-theoretic criterion for determining the equivalence of causal models having different but homomorphic directed acyclic graphs over discrete variables. Following Jacobs et al. (2019), we define a causal model as a probabilistic interpretation of a causal string diagram, i.e., a functor from the ``syntactic'' category $\textsf{Syn}_G$ of graph $G$ to the category $\textsf{Stoch}$ of finite sets and stochastic matrices. The equivalence of causal models is then defined in terms of a natural transformation or isomorphism between two such functors, which we call a $Φ$-abstracti…
Causality knowledge is vital to building robust AI systems. Deep learning models often perform poorly on tasks that require causal reasoning, which is often derived using some form of commonsense knowledge not immediately available in the input but implicitly inferred by humans. Prior work has unraveled spurious observational biases that models fall prey to in the absence of causality. While language representation models preserve contextual knowledge within learned embeddings, they do not factor in causal relationships during training. By blending causal relationships with the input features …
Mechanistic Interpretability of Brain-to-Speech Models Across Speech Modes
Department of Computer Science, University of Maryland, College Park, United States（计算机科学系，马里兰大学，College Park，美国）；Department of Electrical and Computer Engineering, University of Maryland, College Park, United States（电气与计算机工程系，马里兰大学，College Park，美国）
Causality is Key for Interpretability Claims to Generalise
Mila - Qu\'ebec AI Institute \& Universit\'e de Montr\'eal；Boston University；Cold Spring Harbor Laboratory；Max-Planck-Institute for Intelligent Systems, ELLIS Institute T\"ubingen, University of T\"ubingen
Knowledge Abstraction for Knowledge-based Semantic Communication: A Generative Causality Invariant Approach
Department of Intelligent Computing and Data Science, VinUniversity（智能计算与数据科学系，文大学）；School of Computer Science and Statistics, Trinity College Dublin（计算机科学与统计学系，三一学院）；School of Computer Science, The University of Sydney（计算机科学系，悉尼大学）；School of Engineering, The University of Newcastle（工程系，新castle大学）；Department of Information Convergence Engineering, Pusan National University（信息融合工程系，釜山国立大学）
A Multi-Level Causal Intervention Framework for Mechanistic Interpretability in Variational Autoencoders
Indian Institute of Technology, Patna（印度理工学院巴特那分校）；Rajarshi School of Management Technology, Varanasi（拉贾尔希管理技术学院瓦拉纳西分校）；Jaypee Institute of Information Technology, Noida（杰皮信息技术学院诺伊达分校）
A Transition System Abstraction Framework for Neural Network Dynamical System Models
School of Computer and Cyber Sciences, Augusta University（奥古斯塔大学计算机与网络科学学院）；School of Computing, University of Nebraska-Lincoln（内布拉斯加大学林肯分校计算学院）；School of Electrical Engineering, Southwest Jiaotong University（西南交通大学电气工程学院）
Southwest Petroleum University（西南石油大学）；Institute for Artificial Intelligence, Southwest Petroleum University（西南石油大学人工智能研究院）；State Key Laboratory of Oil and Gas Reservoir Geology and Exploitation（油气藏地质及开发工程国家重点实验室）
Machine learning approaches are widely studied in the production prediction of CBM wells after hydraulic fracturing, but merely used in practice due to the low generalization ability and the lack of interpretability. A novel methodology is proposed in this article to discover the latent causality from observed data, which is aimed at finding an indirect way to interpret the machine learning results. Based on the theory of causal discovery, a causal graph is derived with explicit input, output, treatment and confounding variables. Then, SHAP is employed to analyze the influence of the factors o…
