Machine learning holds great promise for advancing the field of medicine, with electronic health records (EHRs) serving as a primary data source. However, EHRs are often sparse and contain missing data due to various challenges and limitations in data collection and sharing between healthcare providers. Existing techniques for imputing missing values predominantly focus on known unknowns, such as missing or unavailable values of lab test results; most do not explicitly address situations where it is difficult to distinguish what is missing. For instance, a missing diagnosis code in an EHR could signify either that the patient has not been diagnosed with the condition or that a diagnosis was made, but not shared by a provider. Such situations fall into the paradigm of unknown unknowns. To address this challenge, we develop a general purpose algorithm for denoising data to recover unknown missing values in binary EHRs. We design a transformer-based denoising neural network where the output is thresholded adaptively to recover values in cases where we predict data are missing. Our results demonstrate improved accuracy in denoising medical codes within a real EHR dataset compared to existing imputation approaches and leads to increased performance on downstream tasks using the denoised data. In particular, when applying our method to a real world application, predicting hospital readmission from EHRs, our method achieves statistically significant improvement over all existing baselines.
Due to the cost or interference of measurement, we need to control measurement system. Assuming that each variable can be measured sequentially, there exists optimal policy choosing next measurement for the former observations. Though optimal measurement policy is actually dependent on the goal of measurement, we mainly focus on retrieving complete data, so called as imputation. Also, we adapt the imputation method to missingness varying with measurement policy. However, learning measurement policy and imputation requires complete data which is impossible to be observed, unfortunately. To tack…
Fed-ReMasker: Federated Tabular Imputation under Feature-Level Missingness
ARTORG Center for Biomedical Engineering Research University of Bern（伯尔尼大学生物医学工程研究中心）；Graduate School for Cellular and Biomedical Sciences University of Bern（伯尔尼大学细胞与生物医学科学研究生院）
Toronto Rehabilitation Institute（多伦多康复研究所）；United Arab Emirates University（阿联酋大学）；University of Toronto（多伦多大学）
Presence of missing values in a dataset can adversely affect the performance of a classifier. Single and Multiple Imputation are normally performed to fill in the missing values. In this paper, we present several variants of combining single and multiple imputation with bootstrapping to create ensembles that can model uncertainty and diversity in the data, and that are robust to high missingness in the data. We present three ensemble strategies: bootstrapping on incomplete data followed by (i) single imputation and (ii) multiple imputation, and (iii) multiple imputation ensemble without bootst…
Tennessee State University（田纳西州立大学）；Old Dominion University（老道明大学）
Missing values in tabular data restrict the use and performance of machine learning, requiring the imputation of missing values. The most popular imputation algorithm is arguably multiple imputations using chains of equations (MICE), which estimates missing values from linear conditioning on observed values. This paper proposes methods to improve both the imputation accuracy of MICE and the classification accuracy of imputed data by replacing MICE's linear conditioning with ensemble learning and deep neural networks (DNN). The imputation accuracy is further improved by characterizing individua…
M$^3$-Impute: Mask-guided Representation Learning for Missing Value Imputation
BNU-HKBU United International College（北师大-浸会大学联合国际学院）；University College Dublin（都柏林大学学院）；The Hong Kong University of Science and Technology（香港科技大学）；Texas State University（德克萨斯州立大学）
University of the Witwatersrand（金山大学）；University of Johannesburg（约翰内斯堡大学）
Missing data is a common concern in health datasets, and its impact on good decision-making processes is well documented. Our study's contribution is a methodology for tackling missing data problems using a combination of synthetic dataset generation, missing data imputation and deep learning methods to resolve missing data challenges. Specifically, we conducted a series of experiments with these objectives; $a)$ generating a realistic synthetic dataset, $b)$ simulating data missingness, $c)$ recovering the missing data, and $d)$ analyzing imputation performance. Our methodology used a gaussia…
PAMF: Prior-Aware Multimodal Fusion for Incomplete Time Series Data
Department of Computer Science, University of Central Florida（中央佛罗里达大学计算机科学系）；Department of Computer Science, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校计算机科学系）
Beyond Random Missingness: Clinically Rethinking for Healthcare Time Series Imputation
King’s College London（伦敦国王学院）；PyPOTS Research（PyPOTS研究中心）；University of Oxford（牛津大学）；Hong Kong University of Science and Technology（香港科技大学）；University College London（伦敦大学学院）
Imputation Strategies Under Clinical Presence: Impact on Algorithmic Fairness
MRC Biostatistics Unit（MRC生物统计学单元）；University of Cambridge（剑桥大学）；The Alan Turing Institute（阿兰·图灵研究所）；McCombs School of Business（麦库姆斯商学院）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Rady School of Management（雷迪管理学院）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Cambridge（剑桥大学）；University of Oxford（牛津大学）；The Alan Turing Institute（艾伦·图灵研究所）
Missing data is an important problem in machine learning practice. Starting from the premise that imputation methods should preserve the causal structure of the data, we develop a regularization scheme that encourages any baseline imputation method to be causally consistent with the underlying data generating mechanism. Our proposal is a causally-aware imputation algorithm (MIRACLE). MIRACLE iteratively refines the imputation of a baseline by simultaneously modeling the missingness generating mechanism, encouraging imputation to be consistent with the causal structure of the data. We conduct e…
Missing Data Multiple Imputation for Tabular Q-Learning in Online RL
Department of Statistics, Harvard University, Cambridge, United States（哈佛大学统计学系）；Department of Statistics, Stanford University, Stanford, United States（斯坦福大学统计学系）；Department of Computer Science, Harvard University, Cambridge, United States（哈佛大学计算机科学系）
Missing data are a concern in many real world data sets and imputation methods are often needed to estimate the values of missing data, but data sets with excessive missingness and high dimensionality challenge most approaches to imputation. Here we show that appropriate feature selection can be an effective preprocessing step for imputation, allowing for more accurate imputation and subsequent model predictions. The key feature of this preprocessing is that it incorporates uncertainty: by accounting for uncertainty due to missingness when selecting features we can reduce the degree of missing…
Department of Computing Science Simon Fraser University（西蒙弗雷泽大学计算机科学系）
Missing data is a significant problem impacting all domains. State-of-the-art framework for minimizing missing data bias is multiple imputation, for which the choice of an imputation model remains nontrivial. We propose a multiple imputation model based on overcomplete deep denoising autoencoders. Our proposed model is capable of handling different data types, missingness patterns, missingness proportions and distributions. Evaluation on several real life datasets show our proposed model significantly outperforms current state-of-the-art methods under varying conditions while simultaneously im…
Duke University（杜克大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Though recent works have developed methods that can generate estimates (or imputations) of the missing entries in a dataset to facilitate downstream analysis, most depend on assumptions that may not align with real-world applications and could suffer from poor performance in subsequent tasks such as classification. This is particularly true if the data have large missingness rates or a small sample size. More importantly, the imputation error could be propagated into the prediction step that follows, which may constrain the capabilities of the prediction model. In this work, we introduce the g…
University of Cambridge（剑桥大学）；UCLA（加州大学洛杉矶分校）；Meta；Alan Turing Institute（艾伦·图灵研究所）
Missing data is a systemic problem in practical scenarios that causes noise and bias when estimating treatment effects. This makes treatment effect estimation from data with missingness a particularly tricky endeavour. A key reason for this is that standard assumptions on missingness are rendered insufficient due to the presence of an additional variable, treatment, besides the input (e.g. an individual) and the label (e.g. an outcome). The treatment variable introduces additional complexity with respect to why some variables are missing that is not fully explored by previous work. In our work…
University of Oxford（牛津大学）；CEREMADE, CNRS Université Paris-Dauphine（巴黎多芬大学CNRS-CEREMADE实验室）
We propose a variational autoencoder architecture to model both ignorable and nonignorable missing data using pattern-set mixtures as proposed by Little (1993). Our model explicitly learns to cluster the missing data into missingness pattern sets based on the observed data and missingness masks. Underpinning our approach is the assumption that the data distribution under missingness is probabilistically semi-supervised by samples from the observed data distribution. Our setup trades off the characteristics of ignorable and nonignorable missingness and can thus be applied to data of both types.…
Most practical data science problems encounter missing data. A wide variety of solutions exist, each with strengths and weaknesses that depend upon the missingness-generating process. Here we develop a theoretical framework for training and inference using only observed variables enabling modeling of incomplete datasets without imputation. Using an information and measure-theoretic argument we construct models with latent space representations that regularize against the potential bias introduced by missing data. The theoretical properties of this approach are demonstrated empirically using a …
One Pipeline, Many Transformers: Pattern-Specific Imputation Specialists for Tabular Missing Data
Industrial Engineering \& Operations Research, Columbia University, New York, USA；Department of Statistics, Columbia University, New York, USA；Operations Research \& Information Engineering, Cornell Tech, New York, USA
Gary and Mary West Health Institute（加里与玛丽·韦斯特健康研究所）
Although data may be abundant, complete data is less so, due to missing columns or rows. This missingness undermines the performance of downstream data products that either omit incomplete cases or create derived completed data for subsequent processing. Appropriately managing missing data is required in order to fully exploit and correctly use data. We propose a Multiple Imputation model using Denoising Autoencoders to learn the internal representation of data. Furthermore, we use the novel mechanisms of Metamorphic Truth and Imputation Feedback to maintain statistical integrity of attributes…
GAUDI: Geometry-Aware Diffusion for Calibrated Air-Quality Time-Series Imputation
Columbia University（哥伦比亚大学）；University of California, Davis（加州大学戴维斯分校）；University of California, San Diego（加州大学圣迭戈分校）；Georgia Institute of Technology（佐治亚理工学院）；Lehigh University（理海大学）；Stony Brook University（纽约州立大学石溪分校）；San Francisco State University（旧金山州立大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加州大学）
University of Washington（华盛顿大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
In wearable sensing applications, data is inevitable to be irregularly sampled or partially missing, which pose challenges for any downstream application. An unique aspect of wearable data is that it is time-series data and each channel can be correlated to another one, such as x, y, z axis of accelerometer. We argue that traditional methods have rarely made use of both times-series dynamics of the data as well as the relatedness of the features from different sensors. We propose a model, termed as DynImp, to handle different time point's missingness with nearest neighbors along feature axis a…
DeepSea Technologies；University of Oxford（牛津大学）；University of Cambridge（剑桥大学）
The problem of missing data, usually absent incurated and competition-standard datasets, is an unfortunate reality for most machine learning models used in industry applications. Recent work has focused on understanding the nature and the negative effects of such phenomena, while devising solutions for optimal imputation of the missing data, using both discriminative and generative approaches. We propose a novel mechanism based on multi-head attention which can be applied effortlessly in any model and achieves better downstream performance without the introduction of the full dataset in any pa…
Increasing Missingness to Reduce Bias: Richardson-SGD with Missing Data
Sorbonne Université and Université Paris Cité, CNRS, Laboratoire de Probabilités, Statistique et Modélisation, LPSM（索邦大学和巴黎cité大学，CNRS，概率、统计与建模实验室，LPSM）
MoCap-Impute: A Comprehensive Benchmark and Comparative Analysis of Imputation Methods for IMU-based Motion Capture Data
Peter Faber Business School, Australian Catholic University（澳大利亚天主教大学彼得·法伯商学院）；University of Technology Sydney（技术与应用科学大学悉尼分校）；Department of Information Technology, College of Computing and Information Sciences, University of Technology and Applied Sciences（技术与应用科学大学信息科技系）；Faculty of Computers and Informatics, Zagazig University（扎基尔大学计算机与信息学院）；AI Applications Research Chair, University of Nizwa（努孜瓦大学人工智能应用研究 chair）；Department of Data Science, Wakeb（数据科学系，Wakeb）；Department of Computer Science, College of Computer Engineering and Sciences, Prince Sattam Bin Abdulaziz University（计算机科学系，普林斯·萨塔姆·本·阿卜杜勒阿齐兹大学）；Higher Future Institute for Specialized Technological Studies, Cairo, Egypt（埃及未来高级专门技术研究学院）；Department of Computer Science, Faculty of Computers & Informatics, Suez Canal University（计算机科学系，苏伊士运河大学）；Department of Mathematics, Faculty of Science, Suez Canal University（数学系，苏伊士运河大学）
Large Language Models for Missing Data Imputation: Understanding Behavior, Hallucination Effects, and Control Mechanisms
Computer Science Division, Aeronautics Institute of Technology（航空技术研究所计算机科学系）；Technology Institute, Federal University of São Paulo（圣保罗联邦大学技术学院）；University of Coimbra, CISUC/LASI – Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering（科英布拉大学）
Stanford School of Medicine（斯坦福大学医学院）；Stanford University（斯坦福大学）
There is an emerging trend in the reinforcement learning for healthcare literature. In order to prepare longitudinal, irregularly sampled, clinical datasets for reinforcement learning algorithms, many researchers will resample the time series data to short, regular intervals and use last-observation-carried-forward (LOCF) imputation to fill in these gaps. Typically, they will not maintain any explicit information about which values were imputed. In this work, we (1) call attention to this practice and discuss its potential implications; (2) propose an alternative representation of the patient …
University of Cambridge（剑桥大学）；Microsoft Research Cambridge（微软剑桥研究院）
Real-world datasets often have missing values associated with complex generative processes, where the cause of the missingness may not be fully observed. This is known as missing not at random (MNAR) data. However, many imputation methods do not take into account the missingness mechanism, resulting in biased imputation values when MNAR data is present. Although there are a few methods that have considered the MNAR scenario, their model's identifiability under MNAR is generally not guaranteed. That is, model parameters can not be uniquely determined even with infinite data samples, hence the i…
Imputation of missing data is a common application in various classification problems where the feature training matrix has missingness. A widely used solution to this imputation problem is based on the lazy learning technique, $k$-nearest neighbor (kNN) approach. However, most of the previous work on missing data does not take into account the presence of the class label in the classification problem. Also, existing kNN imputation methods use variants of Minkowski distance as a measure of distance, which does not work well with heterogeneous data. In this paper, we propose a novel iterative k…
SHIFT: Survival Prediction from Incomplete and Heterogeneous Genomic Data
Technical University of Munich (TUM)（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；School of Medicine and Health Technical University of Munich (TUM)（医学与健康学院慕尼黑技术大学）；Department of Radiation Oncology TUM University Hospital（放射肿瘤科TUM大学医院）；Chair for AI in Healthcare and Medicine Technical University of Munich (TUM)（医疗与医学人工智能系慕尼黑技术大学）；Department of Pathology School of Medicine Department of Biomedical Engineering School of Engineering University of California Irvine（病理学系医学院生物医学工程系工程学院加州大学伊文斯顿分校）
KAIST（韩国科学技术院）；AITRICS（爱特瑞克斯）；Yonsei University College of Medicine（延世大学医学院）
Handling missing data is one of the most fundamental problems in machine learning. Among many approaches, the simplest and most intuitive way is zero imputation, which treats the value of a missing entry simply as zero. However, many studies have experimentally confirmed that zero imputation results in suboptimal performances in training neural networks. Yet, none of the existing work has explained what brings such performance degradations. In this paper, we introduce the variable sparsity problem (VSP), which describes a phenomenon where the output of a predictive model largely varies with re…
Generative Modeling under Non-Monotone MAR Missingness via Approximate Wasserstein Gradient Flows
Department of Mathematics（数学系）；Research Institute for Statistics and Information Science（统计与信息科学研究所）；University of Geneva（日内瓦大学）；University of Hamburg（汉堡大学）
