Missing data has the potential to affect analyses conducted in all fields of scientific study, including healthcare, economics, and the social sciences. Several approaches to unbiased inference in the presence of non-ignorable missingness rely on the specification of the target distribution and its missingness process as a probability distribution that factorizes with respect to a directed acyclic graph. In this paper, we address the longstanding question of the characterization of models that are identifiable within this class of missing data distributions. We provide the first completeness result in this field of study -- necessary and sufficient graphical conditions under which, the full data distribution can be recovered from the observed data distribution. We then simultaneously address issues that may arise due to the presence of both missing data and unmeasured confounding, by extending these graphical conditions and proofs of completeness, to settings where some variables are not just missing, but completely unobserved.
This paper reviews recent advances in missing data research using graphical models to represent multivariate dependencies. We first examine the limitations of traditional frameworks from three different perspectives: \textit{transparency, estimability and testability}. We then show how procedures based on graphical models can overcome these limitations and provide meaningful performance guarantees even when data are Missing Not At Random (MNAR). In particular, we identify conditions that guarantee consistent estimation in broad categories of missing data problems, and derive procedures for imp…
We propose estimation methods for change points in high-dimensional covariance structures with an emphasis on challenging scenarios with missing values. We advocate three imputation like methods and investigate their implications on common losses used for change point detection. We also discuss how model selection methods have to be adapted to the setting of incomplete data. The methods are compared in a simulation study and applied to a time series from an environmental monitoring system. An implementation of our proposals within the R-package hdcd is available via the Supplementary materials…
We propose to learn latent graphical models when data have mixed variables and missing values. This model could be used for further data analysis, including regression, classification, ranking etc. It also could be used for imputing missing values. We specify a latent Gaussian model for the data, where the categorical variables are generated by discretizing an unobserved variable and the latent variables are multivariate Gaussian. The observed data consists of two parts: observed Gaussian variables and observed categorical variables, where the latter part is considered as partially missing Gau…
We introduce the concept of pattern graphs--directed acyclic graphs representing how response patterns are associated. A pattern graph represents an identifying restriction that is nonparametrically identified/saturated and is often a missing not at random restriction. We introduce a selection model and a pattern mixture model formulations using the pattern graphs and show that they are equivalent. A pattern graph leads to an inverse probability weighting estimator as well as an imputation-based estimator. We also study the semi-parametric efficiency theory and derive a multiply-robust estimat…
Graphical network inference is used in many fields such as genomics or ecology to infer the conditional independence structure between variables, from measurements of gene expression or species abundances for instance. In many practical cases, not all variables involved in the network have been observed, and the samples are actually drawn from a distribution where some variables have been marginalized out. This challenges the sparsity assumption commonly made in graphical model inference, since marginalization yields locally dense structures, even when the original network is sparse. We presen…
Model selection and learning the structure of graphical models from the data sample constitutes an important field of probabilistic graphical model research, as in most of the situations the structure is unknown and has to be learnt from the given dataset. In this paper, we present a new forward model selection algorithm for graphical log-linear models. We use mutual conditional independence check to reduce the search space which also takes care of the evaluation of the joint effects and chances of missing important interactions are eliminated. We illustrate our algorithm with a real dataset e…
Harvard University（哈佛大学）；The University of Tokyo（东京大学）；RIKEN Center for Brain Science（理化学研究所脑科学中心）；Iowa State University（爱荷华州立大学）
Several statistical models are given in the form of unnormalized densities, and calculation of the normalization constant is intractable. We propose estimation methods for such unnormalized models with missing data. The key concept is to combine imputation techniques with estimators for unnormalized models including noise contrastive estimation and score matching. In addition, we derive asymptotic distributions of the proposed estimators and construct confidence intervals. Simulation results with truncated Gaussian graphical models and the application to real data of wind direction reveal that…
PURPOSE: Clinical examinations are performed on the basis of necessity. However, our decisions to investigate and document are influenced by various other factors, such as workload and preconceptions. Data missingness patterns may contain insights into conscious and unconscious norms of clinical practice. METHODS: We examine data from the SPOTLIGHT study, a multi-centre cohort study of the effect of prompt ICU admission on mortality. We identify missing values and generate an auxiliary dataset indicating the missing entries. We deploy sparse Gaussian Graphical modelling techniques to identify …
We introduce a self-censoring model for multivariate nonignorable nonmonotone missing data, where the missingness process of each outcome is affected by its own value and is associated with missingness indicators of other outcomes, while conditionally independent of the other outcomes. The self-censoring model complements previous graphical approaches for the analysis of multivariate nonignorable missing data. It is identified under a completeness condition stating that any variability in one outcome can be captured by variability in the other outcomes among complete cases. For estimation, we …
Missing data are frequently encountered in high-dimensional problems, but they are usually difficult to deal with using standard algorithms, such as the expectation-maximization (EM) algorithm and its variants. To tackle this difficulty, some problem-specific algorithms have been developed in the literature, but there still lacks a general algorithm. This work is to fill the gap: we propose a general algorithm for high-dimensional missing data problems. The proposed algorithm works by iterating between an imputation step and a consistency step. At the imputation step, the missing data are impu…
Respondent-driven sampling (RDS) is a chain-referral method for sampling members of a hidden or hard-to-reach population such as sex workers, homeless people, or drug users via their social network. Most methodological work on RDS has focused on inference of population means under the assumption that subjects' network degree determines their probability of being sampled. Criticism of existing estimators is usually focused on missing data: the underlying network is only partially observed, so it is difficult to determine correct sampling probabilities. In this paper, we show that data collected…
University of Michigan（密歇根大学）；University of California, Riverside（加州大学河滨分校）
Estimating conditional dependence graphs and precision matrices are some of the most common problems in modern statistics and machine learning. When data are fully observed, penalized maximum likelihood-type estimators have become standard tools for estimating graphical models under sparsity conditions. Extensions of these methods to more complex settings where data are contaminated with additive or multiplicative noise have been developed in recent years. In these settings, however, the relative performance of different methods is not well understood and algorithmic gaps still exist. In parti…
The Impact of Missing Data on Causal Discovery: A Multicentric Clinical Study
University of Milano - Bicocca（米兰比可卡大学）；F. Hoffmann - La Roche Ltd（罗氏公司）；Fondazione IRCCS Istituto Nazionale dei Tumori（意大利国家肿瘤研究所）；University of Twente（特文特大学）；RadboudUMC（拉德堡德大学医学中心）；Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA)（达勒莫莱人工智能研究所）
Invited Discussion of "Model Uncertainty and Missing Data: An Objective Bayesian Perspective" by Gonzalo García-Donato , María Eugenia Castellanos , Stefano Cabras Alicia Quirós , and Anabel Forte
Linkage maps are used to identify the location of genes responsible for traits and diseases. New sequencing techniques have created opportunities to substantially increase the density of genetic markers. Such revolutionary advances in technology have given rise to new challenges, such as creating high-density linkage maps. Current multiple testing approaches based on pairwise recombination fractions are underpowered in the high-dimensional setting and do not extend easily to polyploid species. We propose to construct linkage maps using graphical models either via a sparse Gaussian copula or a …
Recombinant Inbred Lines derived from divergent parental lines can display extensive segregation distortion and long-range linkage disequilibrium (LD) between distant loci. These genomic signatures are consistent with epistatic selection during inbreeding. Epistatic interactions affect growth and fertility traits or even cause complete lethality. Detecting epistasis is challenging as multiple testing approaches are under-powered and true long-range LD is difficult to distinguish from drift. Here we develop a method for reconstructing an underlying network of genomic signatures of high-dimensio…
Causal inference amid missingness-specific independencies and mechanism shifts
University of Oslo（奥斯陆大学）；Norwegian Institute of Public Health（挪威公共卫生研究所）；University College Dublin（都柏林大学学院）；INTEGREAT – The Norwegian Center for Knowledge-Driven Machine Learning（INTEGREAT – 挪威知识驱动机器学习中心）
The objective is to assess the extent of variation of data quality and completeness of electronic health records and impact on the robustness of risk predictions of incident cardiovascular disease (CVD) using a risk prediction tool that is based on routinely collected data (QRISK3). The study design is a longitudinal cohort study with a setting of 392 general practices (including 3.6 million patients) linked to hospital admission data. Variation in data quality was assessed using Saez stability metrics quantifying outlyingness of each practice. Statistical frailty models evaluated whether accu…
The subject of this paper is the elucidation of effects of actions from causal assumptions represented as a directed graph, and statistical knowledge given as a probability distribution. In particular, we are interested in predicting conditional distributions resulting from performing an action on a set of variables and, subsequently, taking measurements of another set. We provide a necessary and sufficient graphical condition for the cases where such distributions can be uniquely computed from the available information, as well as an algorithm which performs this computation whenever the cond…
