Consider a collection of datasets generated by unknown interventions on an unknown structural causal model $G$. Recently, Bengio et al. (2020) conjectured that among all candidate models, $G$ is the fastest to adapt from one dataset to another, along with promising experiments. Indeed, intuitively $G$ has less mechanisms to adapt, but this justification is incomplete. Our contribution is a more thorough analysis of this hypothesis. We investigate the adaptation speed of cause-effect SCMs. Using convergence rates from stochastic optimization, we justify that a relevant proxy for adaptation speed is distance in parameter space after intervention. Applying this proxy to categorical and normal cause-effect models, we show two results. When the intervention is on the cause variable, the SCM with the correct causal direction is advantaged by a large factor. When the intervention is on the effect variable, we characterize the relative adaptation speed. Surprisingly, we find situations where the anticausal model is advantaged, falsifying the initial hypothesis. Code to reproduce experiments is available at https://github.com/remilepriol/causal-adaptation-speed
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；ETH Zurich（苏黎世联邦理工学院）；KTH Stockholm（斯德哥尔摩KTH皇家理工学院）；DeepMind（深度思维）
Learning models that offer robust out-of-distribution generalization and fast adaptation is a key challenge in modern machine learning. Modelling causal structure into neural networks holds the promise to accomplish robust zero and few-shot adaptation. Recent advances in differentiable causal discovery have proposed to factorize the data generating process into a set of modules, i.e. one module for the conditional distribution of every variable where only causal parents are used as predictors. Such a modular decomposition of knowledge enables adaptation to distributions shifts by only updating…
Sparse Causal Discovery with Generative Intervention for Unsupervised Graph Domain Adaptation
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab（多媒体信息处理国家重点实验室，计算机科学学院，PKU-Anker LLM实验室）；Peking University（北京大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Paul G. Allen School of Computer Science（保罗·G·艾伦计算机科学学院）；University of Washington（华盛顿大学）；Hong Kong University of Science and Technology（香港科技大学）
Fairness of classification and regression has received much attention recently and various, partially non-compatible, criteria have been proposed. The fairness criteria can be enforced for a given classifier or, alternatively, the data can be adapated to ensure that every classifier trained on the data will adhere to desired fairness criteria. We present a practical data adaption method based on quantile preservation in causal structural equation models. The data adaptation is based on a presumed counterfactual model for the data. While the counterfactual model itself cannot be verified experi…
Domain adaptation (DA) arises as an important problem in statistical machine learning when the source data used to train a model is different from the target data used to test the model. Recent advances in DA have mainly been application-driven and have largely relied on the idea of a common subspace for source and target data. To understand the empirical successes and failures of DA methods, we propose a theoretical framework via structural causal models that enables analysis and comparison of the prediction performance of DA methods. This framework also allows us to itemize the assumptions n…
Causal models can compactly and efficiently encode the data-generating process under all interventions and hence may generalize better under changes in distribution. These models are often represented as Bayesian networks and learning them scales poorly with the number of variables. Moreover, these approaches cannot leverage previously learned knowledge to help with learning new causal models. In order to tackle these challenges, we represent a novel algorithm called \textit{causal relational networks} (CRN) for learning causal models using neural networks. The CRN represent causal models usin…
Arizona State University（亚利桑那州立大学）；Army Research Lab（陆军研究实验室）
Studies show that the representations learned by deep neural networks can be transferred to similar prediction tasks in other domains for which we do not have enough labeled data. However, as we transition to higher layers in the model, the representations become more task-specific and less generalizable. Recent research on deep domain adaptation proposed to mitigate this problem by forcing the deep model to learn more transferable feature representations across domains. This is achieved by incorporating domain adaptation methods into deep learning pipeline. The majority of existing models lea…
New York University（纽约大学）；Genentech（基因泰克）；NYU Langone Health（纽约大学朗格尼医学中心）
Domain adaptation and covariate shift are big issues in deep learning and they ultimately affect any causal inference algorithms that rely on deep neural networks. Causal effect variational autoencoder (CEVAE) is trained to predict the outcome given observational treatment data and it suffers from the distribution shift at test time. In this paper, we introduce uniform treatment variational autoencoders (UTVAE) that are trained with uniform treatment distribution using importance sampling and show that using uniform treatment over observational treatment distribution leads to better causal inf…
IBM Research（IBM研究院）；University of Amsterdam（阿姆斯特丹大学）；Radboud University Nijmegen（内梅亨大学）
An important goal common to domain adaptation and causal inference is to make accurate predictions when the distributions for the source (or training) domain(s) and target (or test) domain(s) differ. In many cases, these different distributions can be modeled as different contexts of a single underlying system, in which each distribution corresponds to a different perturbation of the system, or in causal terms, an intervention. We focus on a class of such causal domain adaptation problems, where data for one or more source domains are given, and the task is to predict the distribution of a cer…
Hefei University of Technology（合肥工业大学）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息学院）；Key Laboratory of Knowledge Engineering with Big Data (Hefei University of Technology), Ministry of Education（大数据知识工程教育部重点实验室）；Shanxi University（山西大学）；School of Computer and Information Technology, Shanxi University（山西大学计算机与信息技术学院）；University of South Australia（南澳大学）；UniSA STEM, University of South Australia（南澳大学UniSA STEM学院）
Domain adaptation solves the learning problem in a target domain by leveraging the knowledge in a relevant source domain. While remarkable advances have been made, almost all existing domain adaptation methods heavily require large amounts of unlabeled target domain data for learning domain invariant representations to achieve good generalizability on the target domain. In fact, in many real-world applications, target domain data may not always be available. In this paper, we study the cases where at the training phase the target domain data is unavailable and only well-labeled source domain d…
We study few-shot supervised domain adaptation (DA) for regression problems, where only a few labeled target domain data and many labeled source domain data are available. Many of the current DA methods base their transfer assumptions on either parametrized distribution shift or apparent distribution similarities, e.g., identical conditionals or small distributional discrepancies. However, these assumptions may preclude the possibility of adaptation from intricately shifted and apparently very different distributions. To overcome this problem, we propose mechanism transfer, a meta-distribution…
Applied AI Lab（应用人工智能实验室）；University of Oxford（牛津大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Latent world models allow agents to reason about complex environments with high-dimensional observations. However, adapting to new environments and effectively leveraging previous knowledge remain significant challenges. We present variational causal dynamics (VCD), a structured world model that exploits the invariance of causal mechanisms across environments to achieve fast and modular adaptation. By causally factorising a transition model, VCD is able to identify reusable components across different environments. This is achieved by combining causal discovery and variational inference to lea…
Purdue University（普渡大学）；Kalinga Institute of Industrial Technology（卡林加工业技术学院）；University of South Carolina（南卡罗来纳大学）
One of the most critical problems in transfer learning is the task of domain adaptation, where the goal is to apply an algorithm trained in one or more source domains to a different (but related) target domain. This paper deals with domain adaptation in the presence of covariate shift while invariances exist across domains. One of the main limitations of existing causal inference methods for solving this problem is scalability. To overcome this difficulty, we propose SCTL, an algorithm that avoids an exhaustive search and identifies invariant causal features across source and target domains ba…
How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings?
Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Causal Artificial Intelligence Lab, Columbia University（哥伦比亚大学因果人工智能实验室）；Department of Statistics, Columbia University（哥伦比亚大学统计系）
University of Pittsburgh（匹兹堡大学）；Carnegie Mellon University（卡内基梅隆大学）；The University of Sydney（悉尼大学）
An essential problem in domain adaptation is to understand and make use of distribution changes across domains. For this purpose, we first propose a flexible Generative Domain Adaptation Network (G-DAN) with specific latent variables to capture changes in the generating process of features across domains. By explicitly modeling the changes, one can even generate data in new domains using the generating process with new values for the latent variables in G-DAN. In practice, the process to generate all features together may involve high-dimensional latent variables, requiring dealing with distri…
Consequential decision-making incentivizes individuals to strategically adapt their behavior to the specifics of the decision rule. While a long line of work has viewed strategic adaptation as gaming and attempted to mitigate its effects, recent work has instead sought to design classifiers that incentivize individuals to improve a desired quality. Key to both accounts is a cost function that dictates which adaptations are rational to undertake. In this work, we develop a causal framework for strategic adaptation. Our causal perspective clearly distinguishes between gaming and improvement and …
Université de Montréal（蒙特利尔大学）；Ruprecht-Karls-Universität Heidelberg（海德堡大学）；École Polytechnique Montréal（蒙特利尔综合理工学院）
We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned whil…
Perfect adaptation in a dynamical system is the phenomenon that one or more variables have an initial transient response to a persistent change in an external stimulus but revert to their original value as the system converges to equilibrium. With the help of the causal ordering algorithm, one can construct graphical representations of dynamical systems that represent the causal relations between the variables and the conditional independences in the equilibrium distribution. We apply these tools to formulate sufficient graphical conditions for identifying perfect adaptation from a set of firs…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Univ. of Cambridge（剑桥大学）；Delft Univ. of Technology（代尔夫特理工大学）；Univ. of Copenhagen（哥本哈根大学）
Current methods for covariate-shift adaptation use unlabelled data to compute importance weights or domain-invariant features, while the final model is trained on labelled data only. Here, we consider a particular case of covariate shift which allows us also to learn from unlabelled data, that is, combining adaptation with semi-supervised learning. Using ideas from causality, we argue that this requires learning with both causes, $X_C$, and effects, $X_E$, of a target variable, $Y$, and show how this setting leads to what we call a semi-generative model, $P(Y,X_E|X_C,θ)$. Our approach is robus…
Causal Effect Identification in LiNGAM Models with Latent Confounders
Technical University of Munich（慕尼黑工业大学）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Leiden University（莱顿大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
