Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions typically lack identifiability in parameter space, because they are overparameterized by design. In this paper, building on recent advances in nonlinear ICA, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.
Model identifiability is a desirable property in the context of unsupervised representation learning. In absence thereof, different models may be observationally indistinguishable while yielding representations that are nontrivially related to one another, thus making the recovery of a ground truth generative model fundamentally impossible, as often shown through suitably constructed counterexamples. In this note, we discuss one such construction, illustrating a potential failure case of an identifiability result presented in "Desiderata for Representation Learning: A Causal Perspective" by Wa…
Institute of Data Science（数据科学研究所）；NUS Graduate School for Integrative Sciences and Engineering（新加坡国立大学综合科学与工程研究生院）；National University of Singapore（新加坡国立大学）
Identifiability, or recovery of the true latent representations from which the observed data originates, is de facto a fundamental goal of representation learning. Yet, most deep generative models do not address the question of identifiability, and thus fail to deliver on the promise of the recovery of the true latent sources that generate the observations. Recent work proposed identifiable generative modelling using variational autoencoders (iVAE) with a theory of identifiability. Due to the intractablity of KL divergence between variational approximate posterior and the true posterior, howev…
Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, a line of recent work reveals that contrastive learning can invert the data generating process and recover ground truth latent factors shared between views. In this work, we present new identifiability results for multimodal contrastive learning, showing that it is possible to recover shared factors in a more general setup than the multi-view setting studied previously. Specificall…
Identifiability Guarantees for Causal Disentanglement from Purely Observational Data
Massachusetts Institute of Technology（麻省理工学院）；Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学布罗德研究所）；LIDS, Massachusetts Institute of Technology（麻省理工学院信息与决策系统实验室）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Department of Engineering, University of Cambridge（剑桥大学工程系）；Technical University of Denmark（丹麦技术大学）
In representation learning, a common approach is to seek representations which disentangle the underlying factors of variation. Eastwood & Williams (2018) proposed three metrics for quantifying the quality of such disentangled representations: disentanglement (D), completeness (C) and informativeness (I). In this work, we first connect this DCI framework to two common notions of linear and nonlinear identifiability, thereby establishing a formal link between disentanglement and the closely-related field of independent component analysis. We then propose an extended DCI-ES framework with two ne…
Nonlinear independent component analysis (nICA) aims at recovering statistically independent latent components that are mixed by unknown nonlinear functions. Central to nICA is the identifiability of the latent components, which had been elusive until very recently. Specifically, Hyvärinen et al. have shown that the nonlinearly mixed latent components are identifiable (up to often inconsequential ambiguities) under a generalized contrastive learning (GCL) formulation, given that the latent components are independent conditioned on a certain auxiliary variable. The GCL-based identifiability of …
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；LIDS, Massachusetts Institute of Technology（麻省理工学院信息与决策系统实验室）；Broad Institute of MIT and Harvard（布罗德研究所）
Learning Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity
Institute for Computational & Mathematical Engineering, Stanford University（斯坦福大学计算与数学工程研究所）；Department of Management Science & Engineering, Stanford University（斯坦福大学管理科学与工程系）
Concept Modulation Models: A Unified Framework for Identifiability and Extrapolation
Department of Statistics and Data Science, Carnegie Mellon University（卡内基梅隆大学统计与数据科学系）；Machine Learning Department, Carnegie Mellon University（卡内基梅隆大学机器学习系）
Disentangling Continuous-Time Latent Dynamics: Identifiability of Latent SDEs via Diffusion Shifts
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；The University of Melbourne（墨尔本大学）；Peking University（北京大学）；Carnegie Mellon University（卡内基梅隆大学）
Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research
Max Planck Institute for Intelligent Systems, Tübingen, Germany（马克斯·普朗克智能系统研究所）；ELLIS Institute Tübingen, Germany（图宾根ELLIS研究所）；Tübingen AI Center, Germany（图宾根人工智能中心）；Department of Computer Science, Brown University, Rhode Island, USA（布朗大学计算机科学系）；Cold Spring Harbor Laboratory, Cold Spring Harbor, New York, USA（冷泉港实验室）
Block Disentanglement in CRL: Bridging Identifiability and Visual State Estimation
Rensselaer Polytechnic Institute（伦斯勒理工学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Swiss AI, EPFL（瑞士人工智能，洛桑联邦理工学院）；Google DeepMind（谷歌DeepMind）；University of Delaware（特拉华大学）
Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Guangdong University of Technology（广东工业大学）
Massachusetts Institute of Technology（麻省理工学院）；Microsoft Research（微软研究院）
Recent advances in probabilistic generative modeling have motivated learning Structural Causal Models (SCM) from observational datasets using deep conditional generative models, also known as Deep Structural Causal Models (DSCM). If successful, DSCMs can be utilized for causal estimation tasks, e.g., for answering counterfactual queries. In this work, we warn practitioners about non-identifiability of counterfactual inference from observational data, even in the absence of unobserved confounding and assuming known causal structure. We prove counterfactual identifiability of monotonic generatio…
We introduce Polytopic Matrix Factorization (PMF) as a novel data decomposition approach. In this new framework, we model input data as unknown linear transformations of some latent vectors drawn from a polytope. In this sense, the article considers a semi-structured data model, in which the input matrix is modeled as the product of a full column rank matrix and a matrix containing samples from a polytope as its column vectors. The choice of polytope reflects the presumed features of the latent components and their mutual relationships. As the factorization criterion, we propose the determinan…
Towards Identifiability of Hierarchical Temporal Causal Representation Learning
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Guangdong University of Technology（广东技术大学）；University of California San Diego（加州大学圣地亚哥分校）
The Gatsby Unit UCL（伦敦大学学院盖茨比研究组）；Dept. of CS and HIIT Univ. Helsinki（赫尔辛基大学计算机科学系及赫尔辛基信息技术研究所）；Div. of Info. Sci. NAIST（奈良先端科学技术大学院大学信息科学部）；Univ. Cambridge & Microsoft Research（剑桥大学及微软研究院）
Nonlinear ICA is a fundamental problem for unsupervised representation learning, emphasizing the capacity to recover the underlying latent variables generating the data (i.e., identifiability). Recently, the very first identifiability proofs for nonlinear ICA have been proposed, leveraging the temporal structure of the independent components. Here, we propose a general framework for nonlinear ICA, which, as a special case, can make use of temporal structure. It is based on augmenting the data by an auxiliary variable, such as the time index, the history of the time series, or any other availab…
Why Conclusions Diverge from the Same Observations: Formalizing World-Model Non-Identifiability via an Inference
Human Informatics and Systems Laboratory, Doshisha University, Kyoto, Japan（大阪大学人文学与系统实验室，京都，日本）；Linked Open Data Initiative, NPO, Tokyo, Japan（开放数据倡议，东京，日本）；Keio Research Institute at SFC, Fujisawa, Japan（庆应义塾大学SFC研究所， Fujisawa，日本）；Stroly Inc., Kyoto, Japan（Stroly公司，京都，日本）
An Interventional Perspective on Identifiability in Gaussian LTI Systems with Independent Component Analysis
Carnegie Mellon University（卡内基梅隆大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；International Max Planck Research School for Intelligent Systems (IMPRS-IS)（马克斯·普朗克智能系统国际研究学校（IMPRS-IS））；European Laboratory for Learning and Intelligent Systems (ELLIS)（欧洲学习与智能系统实验室（ELLIS））
On the Identification of Temporally Causal Representation with Instantaneous Dependence
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Guangdong University of Technology（广东技术大学）；The University of Melbourne（墨尔本大学）
From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signals
School of Computer Science, Guangdong University of Technology（广东工业大学计算机学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Shantou University（汕头大学）；Carnegie Mellon University（卡内基梅隆大学）
Recently in the field of unsupervised representation learning, strong identifiability results for disentanglement of causally-related latent variables have been established by exploiting certain side information, such as class labels, in addition to independence. However, most existing work is constrained by functional form assumptions such as independent sources or further with linear transitions, and distribution assumptions such as stationary, exponential family distribution. It is unknown whether the underlying latent variables and their causal relations are identifiable if they have arbit…
Oregon State Univ.（俄勒冈州立大学）；Google Inc.（谷歌公司）；IBM Research（IBM研究院）
Multiple views of data, both naturally acquired (e.g., image and audio) and artificially produced (e.g., via adding different noise to data samples), have proven useful in enhancing representation learning. Natural views are often handled by multiview analysis tools, e.g., (deep) canonical correlation analysis [(D)CCA], while the artificial ones are frequently used in self-supervised learning (SSL) paradigms, e.g., BYOL and Barlow Twins. Both types of approaches often involve learning neural feature extractors such that the embeddings of data exhibit high cross-view correlations. Although intu…
University College London（伦敦大学学院）；The Alan Turing Institute（艾伦·图灵研究所）
In this paper, we investigate the algorithmic stability of unsupervised representation learning with deep generative models, as a function of repeated re-training on the same input data. Algorithms for learning low dimensional linear representations -- for example principal components analysis (PCA), or linear independent components analysis (ICA) -- come with guarantees that they will always reveal the same latent representations (perhaps up to an arbitrary rotation or permutation). Unfortunately, for non-linear representation learning, such as in a variational auto-encoder (VAE) model traine…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Cambridge（剑桥大学）
Independent component analysis provides a principled framework for unsupervised representation learning, with solid theory on the identifiability of the latent code that generated the data, given only observations of mixtures thereof. Unfortunately, when the mixing is nonlinear, the model is provably nonidentifiable, since statistical independence alone does not sufficiently constrain the problem. Identifiability can be recovered in settings where additional, typically observed variables are included in the generative process. We investigate an alternative path and consider instead including a…
Multi-View Causal Representation Learning with Partial Observability
Institute of Science and Technology Austria（奥地利科学技术研究所）；Max Planck Institute for Intelligent Systems, Tübingen（图宾根马克斯·普朗克智能系统研究所）；University of Amsterdam（阿姆斯特丹大学）；Samsung - SAIT AI Lab（三星SAIT AI实验室）；Mila, Université de Montréal（蒙特利尔大学Mila研究所）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；ServiceNow Research（ServiceNow研究院）；University of Tübingen（图宾根大学）；University of Cambridge（剑桥大学）
As more decisions in our daily life become automated, the need to have machine learning algorithms that make fair decisions increases. In fair representation learning we are tasked with finding a suitable representation of the data in which a sensitive variable is censored. Recent work aims to learn fair representations through adversarial learning. This paper builds upon this work by introducing a novel algorithm which uses dampening and stacking to learn adversarial fair representations. Results show that that our algorithm improves upon earlier work in both censoring and reconstruction.
When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective
Technical University of Denmark（丹麦技术大学）；University of Trento, Italy（意大利特伦托大学）；Helmholtz AI, Munich（慕尼黑海德堡人工智能研究所）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Copenhagen（哥本哈根大学）
