We propose a variational autoencoder architecture to model both ignorable and nonignorable missing data using pattern-set mixtures as proposed by Little (1993). Our model explicitly learns to cluster the missing data into missingness pattern sets based on the observed data and missingness masks. Underpinning our approach is the assumption that the data distribution under missingness is probabilistically semi-supervised by samples from the observed data distribution. Our setup trades off the characteristics of ignorable and nonignorable missingness and can thus be applied to data of both types. We evaluate our method on a wide range of data sets with different types of missingness and achieve state-of-the-art imputation performance. Our model outperforms many common imputation algorithms, especially when the amount of missing data is high and the missingness mechanism is nonignorable.
We are concerned in clustering continuous data sets subject to non-ignorable missingness. We perform clustering with a specific semi-parametric mixture, under the assumption of conditional independence given the component. The mixture model isused for clustering and not for estimating the density of the full variables (observed and unobserved), thus we do not need other assumptions on the component distribution neither to specify the missingness mechanism. Estimation is performed by maximizing an extension of smoothed likelihood allowing missingness. This optimization is achieved by a Majoriza…
Missing data problems arise in many applied research studies. They may jeopardize statistical inference of the model of interest, if the missing mechanism is nonignorable, that is, the missing mechanism depends on the missing values themselves even conditional on the observed data. With a nonignorable missing mechanism, the model of interest is often not identifiable without imposing further assumptions. We find that even if the missing mechanism has a known parametric form, the model is not identifiable without specifying a parametric outcome distribution. Although it is fundamental for valid…
Missingness is a common occurrence in educational assessment and psychological measurement. It could not be casually ignored as it may threaten the validity of the test if not handled properly. Considering the difference between omitted and not-reached items, we developed an IRT-based model to handle these missingness. In the proposed method, not-reached responses are captured by the cumulative missingness. Moreover, the nonignorability is attributed to the correlation between ability and person missing trait. We proved that its item parameters estimate under maximum marginal likelihood (MML) …
Pattern-mixture models provide a transparent approach for handling missing data, where the full-data distribution is factorized in a way that explicitly shows the parts that can be estimated from observed data alone, and the parts that require identifying restrictions. We introduce a nonparametric estimator of the full-data distribution based on the pattern-mixture model factorization. Our approach uses the empirical observed-data distribution and augments it with a nonparametric estimator of the missing-data distributions under a given identifying restriction. Our results apply to a large cla…
A major practical impediment when implementing adaptive dose-finding designs is that the toxicity outcome used by the decision rules may not be observed shortly after the initiation of the treatment. To address this issue, we propose the data augmentation continual reassessment method (DA-CRM) for dose finding. By naturally treating the unobserved toxicities as missing data, we show that such missing data are nonignorable in the sense that the missingness depends on the unobserved outcomes. The Bayesian data augmentation approach is used to sample both the missing data and model parameters fro…
When conducting a paired $2\times2$ crossover design, each subject is paired with another subject with similar characteristics. The pair is then randomized to the same sequence of two treatments. That is, the two subjects receive the first experimental treatment, and then they cross over and receive the other experimental treatment(s). The paired $2\times2$ crossover design that was used in the Beta Adrenergic Response by GEnotype (BARGE) Study conducted by the National Heart, Lung and Blood Institute's Asthma Clinical Research Network (ACRN) has been described elsewhere. When the data arising…
Identifying expanding TCR clonotypes with a longitudinal Bayesian mixture model and their associations with cancer patient prognosis, metastasis-directed therapy, and VJ gene enrichment
Department of Biostatistics（生物统计系）；Department of Radiation Oncology, Mayo Clinic（梅奥诊所放射肿瘤科）；Department of Translational Molecular Pathology（转化分子病理学系）；Departments of Thoracic/Head and Neck Medical Oncology（胸部/头颈医学肿瘤科）；Department of Radiation Oncology, University of Texas MD Anderson Cancer Center（德克萨斯大学安德森癌症中心放射肿瘤科）
UiT The Arctic University of Norway（挪威北极大学）；University Hospital of North Norway（北挪威大学医院）；Universidad Rey Juan Carlos（胡安·卡洛斯国王大学）
The time series cluster kernel (TCK) provides a powerful tool for analysing multivariate time series subject to missing data. TCK is designed using an ensemble learning approach in which Bayesian mixture models form the base models. Because of the Bayesian approach, TCK can naturally deal with missing values without resorting to imputation and the ensemble strategy ensures robustness to hyperparameters, making it particularly well suited for unsupervised learning. However, TCK assumes missing at random and that the underlying missingness mechanism is ignorable, i.e. uninformative, an assumptio…
Partially recorded data are frequently encountered in many applications and usually clustered by first removing incomplete cases or features with missing values, or by imputing missing values, followed by application of a clustering algorithm to the resulting altered dataset. Here, we develop clustering methodology through a model-based approach using the marginal density for the observed values, assuming a finite mixture model of multivariate $t$ distributions. We compare our approximate algorithm to the corresponding full expectation-maximization (EM) approach that considers the missing valu…
In electronic health records (EHRs), latent subgroups of patients may exhibit distinctive patterning in their longitudinal health trajectories. For such data, growth mixture models (GMMs) enable classifying patients into different latent classes based on individual trajectories and hypothesized risk factors. However, the application of GMMs is hindered by the special missing data problem in EHRs, which manifests two patient-led missing data processes: the visit process and the response process for an EHR variable conditional on a patient visiting the clinic. If either process is associated wit…
Informative missingness and its implications in semi-supervised learning
School of Mathematics and Physics, The University of Queensland, Australia（昆士兰大学数学与物理学院）；School of Statistics and Data Science, Guangdong University of Finance & Economics, China（广东财经大学统计与数据科学学院）；School of Mathematics and Computational Science/Hunan Key Laboratory for Computation and Simulation in Science and Engineering, Xiangtan University, China（湘潭大学数学与计算科学学院/湖南计算与工程模拟重点实验室）
We describe the Bedside Patient Rescue (BPR) project, the goal of which is risk prediction of adverse events for non-ICU patients using ~200 variables (vitals, lab results, assessments, ...). There are several missing predictor values for most patients, which in the health sciences is the norm, rather than the exception. A Bayesian approach is presented that addresses many of the shortcomings to standard approaches to missing predictors: (i) treatment of the uncertainty due to imputation is straight-forward in the Bayesian paradigm, (ii) the predictor distribution is flexibly modeled as an inf…
ASELSAN Research Center（ASELSAN研究中心）；Bilkent University（比尔肯特大学）；DataBoss A.S.（DataBoss公司）
We investigate regression for variable length sequential data containing missing samples and introduce a novel tree architecture based on the Long Short-Term Memory (LSTM) networks. In our architecture, we employ a variable number of LSTM networks, which use only the existing inputs in the sequence, in a tree-like architecture without any statistical assumptions or imputations on the missing data, unlike all the previous approaches. In particular, we incorporate the missingness information by selecting a subset of these LSTM networks based on "presence-pattern" of a certain number of previous …
Zero Inflation as a Missing Data Problem: a Proxy-based Approach
Johns Hopkins Whiting School of Engineering（约翰斯·霍普金斯大学惠廷工程学院）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）；Johns Hopkins Bloomberg School of Public Health（约翰斯·霍普金斯大学布隆伯格公共卫生学院）；Johns Hopkins Medicine（约翰斯·霍普金斯医学）；Johns Hopkins Health System（约翰斯·霍普金斯卫生系统）；Vassar Brothers Medical Center（瓦萨兄弟医疗中心）
When Pattern-by-Pattern Works: Theoretical and Empirical Insights for Logistic Models with Missing Values
Department of Statistics, University of Oxford, Oxford, United Kingdom（牛津大学统计学系）；INRIA PreMeDICaL, Institut Desbrest d’Épidémiologie et de Santé Publique (Idesp), University of Montpellier, Montpellier, France（INRIA PreMeDICaL、蒙彼利埃大学）；Sorbonne Université, Université Paris Cité, CNRS, Laboratoire de Probabilités, Statistique et Modélisation (LPSM), F-75005 Paris, France（索邦大学、巴黎城市大学、国家科学研究中心）
