While hidden class models of various types arise in many statistical applications, it is often difficult to establish the identifiability of their parameters. Focusing on models in which there is some structure of independence of some of the observed variables conditioned on hidden ones, we demonstrate a general approach for establishing identifiability utilizing algebraic arguments. A theorem of J. Kruskal for a simple latent-class model with finite state space lies at the core of our results, though we apply it to a diverse set of models. These include mixtures of both finite and nonparametric product distributions, hidden Markov models and random graph mixture models, and lead to a number of new results and improvements to old ones. In the parametric setting, this approach indicates that for such models, the classical definition of identifiability is typically too strong. Instead generic identifiability holds, which implies that the set of nonidentifiable parameters has measure zero, so that parameter inference is still meaningful. In particular, this sheds light on the properties of finite mixtures of Bernoulli products, which have been used for decades despite being known to have nonidentifiable parameters. In the nonparametric setting, we again obtain identifiability only when certain restrictions are placed on the distributions that are mixed, but we explicitly describe the conditions.
Latent class models have wide applications in social and biological sciences. In many applications, pre-specified restrictions are imposed on the parameter space of latent class models, through a design matrix, to reflect practitioners' assumptions about how the observed responses depend on subjects' latent traits. Though widely used in various fields, such restricted latent class models suffer from non-identifiability due to their discreteness nature and complex structure of restrictions. This work addresses the fundamental identifiability issue of restricted latent class models by developing…
We consider linear structural equation models with latent variables and develop a criterion to certify whether the direct causal effects between the observable variables are identifiable based on the observed covariance matrix. Linear structural equation models assume that both observed and latent variables solve a linear equation system featuring stochastic noise terms. Each model corresponds to a directed graph whose edges represent the direct effects that appear as coefficients in the equation system. Prior research has developed a variety of methods to decide identifiability of direct effe…
Latent class models have recently become popular for multiple-systems estimation in human rights applications. However, it is currently unknown when a given family of latent class models is identifiable in this context. We provide necessary and sufficient conditions on the number of latent classes needed for a family of latent class models to be identifiable. Along the way we provide a mechanism for verifying identifiability in a class of multiple-systems estimation models that allow for individual heterogeneity.
Identifiability of parameters is an essential property for a statistical model to be useful in most settings. However, establishing parameter identifiability for Bayesian networks with hidden variables remains challenging. In the context of finite state spaces, we give algebraic arguments establishing identifiability of some special models on small DAGs. We also establish that, for fixed state spaces, generic identifiability of parameters depends only on the Markov equivalence class of the DAG. To illustrate the use of these results, we investigate identifiability for all binary Bayesian netwo…
Restricted latent class models (RLCMs) have recently gained prominence in educational assessment, psychiatric evaluation, and medical diagnosis. Different from conventional latent class models, restrictions on the RLCM model parameters are imposed by a design matrix to respect practitioners' scientific assumptions. The design matrix, called $Q$-matrix in the cognitive diagnosis literature, is usually constructed by practitioners and domain experts, yet it is subjective and could be misspecified. To address this problem, researchers have proposed to estimate the $Q$-matrix from data. On the oth…
High dimensional categorical data are routinely collected in biomedical and social sciences. It is of great importance to build interpretable parsimonious models that perform dimension reduction and uncover meaningful latent structures from such discrete data. Identifiability is a fundamental requirement for valid modeling and inference in such scenarios, yet is challenging to address when there are complex latent structures. In this article, we propose a class of identifiable multilayer (potentially deep) discrete latent structure models for discrete data, termed Bayesian pyramids. We establi…
The bifactor model and its extensions are multidimensional latent variable models, under which each item measures up to one subdimension on top of the primary dimension(s). Despite their wide applications to educational and psychological assessments, this type of multidimensional latent variable models may suffer from non-identifiability, which can further lead to inconsistent parameter estimation and invalid inference. The current work provides a relatively complete characterization of identifiability for the linear and dichotomous bifactor models and the linear extended bifactor model with c…
Statistical latent class models are widely used in social and psychological researches, yet it is often difficult to establish the identifiability of the model parameters. In this paper we consider the identifiability issue of a family of restricted latent class models, where the restriction structures are needed to reflect pre-specified assumptions on the related assessment. We establish the identifiability results in the strict sense and specify which types of restriction structure would give the identifiability of the model parameters. The results not only guarantee the validity of many of …
The problem of identifiability of finite mixtures of finite product measures is studied. A mixture model with $K$ mixture components and $L$ observed variables is considered, where each variable takes its value in a finite set with cardinality $M$.The variables are independent in each mixture component. The identifiability of a mixture model means the possibility of attaining the mixture components parameters by observing its mixture distribution. In this paper, we investigate fundamental relations between the identifiability of mixture models and the separability of their observed variables b…
Columbia University（哥伦比亚大学）；University of Michigan（密歇根大学）
Hierarchical Latent Attribute Models (HLAMs) are a family of discrete latent variable models that are attracting increasing attention in educational, psychological, and behavioral sciences. The key ingredients of an HLAM include a binary structural matrix and a directed acyclic graph specifying hierarchical constraints on the configurations of latent attributes. These components encode practitioners' design information and carry important scientific meanings. Despite the popularity of HLAMs, the fundamental identifiability issue remains unaddressed. The existence of the attribute hierarchy gra…
University of Chicago（芝加哥大学）；Carnegie Mellon University（卡内基梅隆大学）
We prove identifiability of a broad class of deep latent variable models that (a) have universal approximation capabilities and (b) are the decoders of variational autoencoders that are commonly used in practice. Unlike existing work, our analysis does not require weak supervision, auxiliary information, or conditioning in the latent space. Specifically, we show that for a broad class of generative (i.e. unsupervised) models with universal approximation capabilities, the side information $u$ is not necessary: We prove identifiability of the entire generative model where we do not observe $u$ a…
John A. Paulson School of Engineering and Applied Sciences（约翰·A·保尔森工程与应用科学学院）；Harvard University（哈佛大学）
Bayesian Neural Networks with Latent Variables (BNN+LVs) capture predictive uncertainty by explicitly modeling model uncertainty (via priors on network weights) and environmental stochasticity (via a latent input noise variable). In this work, we first show that BNN+LV suffers from a serious form of non-identifiability: explanatory power can be transferred between the model parameters and latent variables while fitting the data equally well. We demonstrate that as a result, in the limit of infinite data, the posterior mode over the network weights and latent variables is asymptotically biased …
We prove identifiability of parameters for a broad class of random graph mixture models. These models are characterized by a partition of the set of graph nodes into latent (unobservable) groups. The connectivities between nodes are independent random variables when conditioned on the groups of the nodes being connected. In the binary random graph case, in which edges are either present or absent, these models are known as stochastic blockmodels and have been widely used in the social sciences and, more recently, in biology. Their generalizations to weighted random graphs, either in parametric…
Concept Modulation Models: A Unified Framework for Identifiability and Extrapolation
Department of Statistics and Data Science, Carnegie Mellon University（卡内基梅隆大学统计与数据科学系）；Machine Learning Department, Carnegie Mellon University（卡内基梅隆大学机器学习系）
Most modern probabilistic generative models, such as the variational autoencoder (VAE), have certain indeterminacies that are unresolvable even with an infinite amount of data. Different tasks tolerate different indeterminacies, however recent applications have indicated the need for strongly identifiable models, in which an observation corresponds to a unique latent code. Progress has been made towards reducing model indeterminacies while maintaining flexibility, and recent work excludes many--but not all--indeterminacies. In this work, we motivate model-identifiability in terms of task-ident…
Prediction Markets as Bayesian Inverse Problems: Uncertainty Quantification, Identifiability, and Information Gain from Price-Volume Histories under Latent Types
We study parameter identifiability of directed Gaussian graphical models with one latent variable. In the scenario we consider, the latent variable is a confounder that forms a source node of the graph and is a parent to all other nodes, which correspond to the observed variables. We give a graphical condition that is sufficient for the Jacobian matrix of the parametrization map to be full rank, which entails that the parametrization is generically finite-to-one, a fact that is sometimes also referred to as local identifiability. We also derive a graphical condition that is necessary for such …
Identification of Nonlinear Latent Hierarchical Models
Carnegie Mellon University（卡内基梅隆大学）；University of California San Diego（加州大学圣迭戈分校）；Beijing Technology and Business University（北京工商大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
The condition of parameter identifiability is essential for the consistency of all estimators and is often challenging to prove. As a consequence, this condition is often assumed for simplicity although this may not be straightforward to assume for a variety of model settings. In this paper we deal with a particular class of models that we refer to as "latent" models which can be defined as models made by the sum of underlying models, such as a variety of linear state-space models for time series. These models are of great importance in many fields, from ecology to engineering, and in this pap…
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学博德研究所）
Structured latent attribute models (SLAMs) are a special family of discrete latent variable models widely used in social and biological sciences. This paper considers the problem of learning significant attribute patterns from a SLAM with potentially high-dimensional configurations of the latent attributes. We address the theoretical identifiability issue, propose a penalized likelihood method for the selection of the attribute patterns, and further establish the selection consistency in such an overfitted SLAM with diverging number of latent patterns. The good performance of the proposed meth…
Beyond Degree: Rooted Motif Signatures for Latent Position Identifiability in Graphon Models
Sorbonne Université（索邦大学）；Université Paris Cité（巴黎西岱大学）；CNRS（法国国家科学研究中心）；Laboratoire de Probabilités, Statistique et Modélisation（概率、统计与建模实验室）；MIA AgroParisTech（农业巴黎高等师范学院数学与信息科学研究所）；INRAE（法国国家农业、食品与环境研究院）；Université Paris-Saclay（巴黎萨克雷大学）
This paper considers hidden Markov models where the observations are given as the sum of a latent state which lies in a general state space and some independent noise with unknown distribution. It is shown that these fully nonparametric translation models are identifiable with respect to both the distribution of the latent variables and the distribution of the noise, under mostly a light tail assumption on the latent variables. Two nonparametric estimation methods are proposed and we prove that the corresponding estimators are consistent for the weak convergence topology. These results are ill…
Synthetic Potential Outcomes and Causal Mixture Identifiability
Eric and Wendy Schmidt Center（埃里克和温迪·施密特中心）；Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学布罗德研究所）；Laboratory for Information and Decision Systems（信息与决策系统实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
This paper outlines a new nonparametric estimation procedure for unobserved phi-mixing processes. It is assumed that the only information on the stationary hidden states (Xk) is given by the process (Yk), where Yk is a noisy observation of f(Xk). The paper introduces a maximum pseudo-likelihood procedure to estimate the function f and the distribution of the hidden states using blocks of observations of length b. The identifiability of the model is studied in the particular cases b=1 and b=2. The consistency of the estimators of f and of the distribution of the hidden states as the number of o…
Identifiability of statistical models is a fundamental and essential condition that is required to prove the consistency of maximum likelihood estimators. The identifiability of the skew families of distributions on the circle and cylinder for estimating model parameters has not been fully investigated in the literature. In this paper, a new method combining the trigonometric moments and the simultaneous Diophantine approximation is proposed to prove the identifiability of asymmetric circular and cylindrical distributions. Using this method, we prove the identifiability of general sine-skewed …
Identifiability Guarantees for Causal Disentanglement from Soft Interventions
Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学布罗德研究所）；Laboratory for Information and Decision Systems, MIT（麻省理工学院信息与决策系统实验室）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；Google Research（谷歌研究院）
We consider structural equation models in which variables can be written as a function of their parents and noise terms, which are assumed to be jointly independent. Corresponding to each structural equation model, there is a directed acyclic graph describing the relationships between the variables. In Gaussian structural equation models with linear functions, the graph can be identified from the joint distribution only up to Markov equivalence classes, assuming faithfulness. In this work, we prove full identifiability if all noise variables have the same variances: the directed acyclic graph …
Nonnegative matrix factorizations and related compositional models: Equivalence, identifiability, and an application on the grain-size analysis of sediments
Hangzhou Dianzi University, China（杭州电子科技大学）；Utrecht University, the Netherlands and University of Southampton, UK（乌得勒支大学和南安普顿大学）
We give a robust version of the celebrated result of Kruskal on the uniqueness of tensor decompositions: we prove that given a tensor whose decomposition satisfies a robust form of Kruskal's rank condition, it is possible to approximately recover the decomposition if the tensor is known up to a sufficiently small (inverse polynomial) error. Kruskal's theorem has found many applications in proving the identifiability of parameters for various latent variable models and mixture models such as Hidden Markov models, topic models etc. Our robust version immediately implies identifiability using onl…
