Multiple Imputation with Denoising Autoencoder using Metamorphic Truth and Imputation Feedback
Gary and Mary West Health Institute（加里与玛丽·韦斯特健康研究所）
Haw-minn Lu, Giancarlo Perrone, José Unpingco
备注：
Machine Learning and Data Mining in Pattern Recognition, 16th International Conference on Machine Learning and Data Mining, MLDM 2020, Amsterdam, The Netherlands, July 20-21, 2020, Proceedings, pp. 197-208
Although data may be abundant, complete data is less so, due to missing columns or rows. This missingness undermines the performance of downstream data products that either omit incomplete cases or create derived completed data for subsequent processing. Appropriately managing missing data is required in order to fully exploit and correctly use data. We propose a Multiple Imputation model using Denoising Autoencoders to learn the internal representation of data. Furthermore, we use the novel mechanisms of Metamorphic Truth and Imputation Feedback to maintain statistical integrity of attributes and eliminate bias in the learning process. Our approach explores the effects of imputation on various missingness mechanisms and patterns of missing data, outperforming other methods in many standard test cases.
Department of Computing Science Simon Fraser University（西蒙弗雷泽大学计算机科学系）
Missing data is a significant problem impacting all domains. State-of-the-art framework for minimizing missing data bias is multiple imputation, for which the choice of an imputation model remains nontrivial. We propose a multiple imputation model based on overcomplete deep denoising autoencoders. Our proposed model is capable of handling different data types, missingness patterns, missingness proportions and distributions. Evaluation on several real life datasets show our proposed model significantly outperforms current state-of-the-art methods under varying conditions while simultaneously im…
Dealing with missing data in data analysis is inevitable. Although powerful imputation methods that address this problem exist, there is still much room for improvement. In this study, we examined single imputation based on deep autoencoders, motivated by the apparent success of deep learning to efficiently extract useful dataset features. We have developed a consistent framework for both training and imputation. Moreover, we benchmarked the results against state-of-the-art imputation methods on different data sizes and characteristics. The work was not limited to the one-type variable dataset…
SunPower Corporation（SunPower公司）；SolarEdge Technologies（SolarEdge科技公司）；Case Western Reserve University（凯斯西储大学）；C2 Energy Capital（C2能源资本公司）；Sandia National Labs（桑迪亚国家实验室）
The integration of the global Photovoltaic (PV) market with real time data-loggers has enabled large scale PV data analytical pipelines for power forecasting and long-term reliability assessment of PV fleets. Nevertheless, the performance of PV data analysis heavily depends on the quality of PV timeseries data. This paper proposes a novel Spatio-Temporal Denoising Graph Autoencoder (STD-GAE) framework to impute missing PV Power Data. STD-GAE exploits temporal correlation, spatial coherence, and value dependencies from domain knowledge to recover missing data. Experimental results show that STD…
Detection and Imputation based Two-Stage Denoising Diffusion Power System Measurement Recovery under Cyber-Physical Uncertainties
State Key Laboratory of Advanced Electromagnetic Technology, School of Electrical and Electronic Engineering, Huazhong University of Science and Technology（华中科技大学电气与电子工程学院先进电磁技术国家重点实验室）；Department of Electrical Engineering and Computer Science, Lassonde School of Engineering, York University（约克大学拉森德工程学院电气工程与计算机科学系）
DiffImpute: Tabular Data Imputation With Denoising Diffusion Probabilistic Model
Spin-Quantum Co. LTD（自旋量子有限公司）；School of Mathematics and Statistics, University of New South Wales（新南威尔士大学数学与统计学院）；Department of Computer Science, Johns Hopkins University（约翰霍普金斯大学计算机科学系）
Missing data is a recurrent and challenging problem, especially when using machine learning algorithms for real-world applications. For this reason, missing data imputation has become an active research area, in which recent deep learning approaches have achieved state-of-the-art results. We propose DAEMA (Denoising Autoencoder with Mask Attention), an algorithm based on a denoising autoencoder architecture with an attention mechanism. While most imputation algorithms use incomplete inputs as they would use complete data - up to basic preprocessing (e.g. mean imputation) - DAEMA leverages a ma…
Missing data imputation for noisy time-series data and applications in healthcare
Faculty of Mathematics and Computer Science, University of Science（胡志明市科学大学数学与计算机科学学院）；Vietnam National University（越南国家大学）；University of Medicine and Pharmacy at Ho Chi Minh City（胡志明市医药大学）；Simula Metropolitan（西穆拉都会研究院）；AISIA Research Laboratory（AISIA研究院）
University of Missouri（密苏里大学）；Meta Platforms, Inc.（元平台公司）；Harvard Medical School（哈佛医学院）；Duke University（杜克大学）
Multiple imputation (MI) is a popular approach for dealing with missing data arising from non-response in sample surveys. Multiple imputation by chained equations (MICE) is one of the most widely used MI algorithms for multivariate data, but it lacks theoretical foundation and is computationally intensive. Recently, missing data imputation methods based on deep learning models have been developed with encouraging results in small studies. However, there has been limited research on evaluating their performance in realistic settings compared to MICE, particularly in big surveys. We conduct exte…
Missing data is one of the most common preprocessing problems. In this paper, we experimentally research the use of generative and non-generative models for feature reconstruction. Variational Autoencoder with Arbitrary Conditioning (VAEAC) and Generative Adversarial Imputation Network (GAIN) were researched as representatives of generative models, while the denoising autoencoder (DAE) represented non-generative models. Performance of the models is compared to traditional methods k-nearest neighbors (k-NN) and Multiple Imputation by Chained Equations (MICE). Moreover, we introduce WGAIN as the…
Opening the Black Box: Towards inherently interpretable energy data imputation models using building physics insight
E3D - Institute of Energy Efficiency and Sustainable Building, RWTH Aachen University（亚琛工业大学E3D能源效率与可持续建筑研究所）；Future Cities Laboratory Global, Singapore-ETH Centre（新加坡-ETH中心未来城市实验室全球分部）；Department of the Built Environment, College of Design and Engineering, National University of Singapore (NUS)（新加坡国立大学设计与工程学院建筑环境系）
Georgia Institute of Technology（佐治亚理工学院）；Anonymous University（匿名大学）
Incomplete instances with various missing attributes in many real-world applications have brought challenges to the classification tasks. Missing values imputation methods are often employed to replace the missing values with substitute values. However, this process often separates the imputation and classification, which may lead to inferior performance since label information are often ignored during imputation. Moreover, traditional methods may rely on improper assumptions to initialize the missing values, whereas the unreliability of such initialization might lead to inferior performance. …
Cross-Domain Conditional Diffusion Models for Time Series Imputation
Northwestern University（西北大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Arizona State University（亚利桑那州立大学）；Virginia Tech（弗吉尼亚理工学院）；Squirrel Ai Learning
Missing data imputation (MDI) is a fundamental problem in many scientific disciplines. Popular methods for MDI use global statistics computed from the entire data set (e.g., the feature-wise medians), or build predictive models operating independently on every instance. In this paper we propose a more general framework for MDI, leveraging recent work in the field of graph neural networks (GNNs). We formulate the MDI task in terms of a graph denoising autoencoder, where each edge of the graph encodes the similarity between two patterns. A GNN encoder learns to build intermediate representations…
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Cornell University（康奈尔大学）
Low dimensional nonlinear structure abounds in datasets across computer vision and machine learning. Kernelized matrix factorization techniques have recently been proposed to learn these nonlinear structures for denoising, classification, dictionary learning, and missing data imputation, by observing that the image of the matrix in a sufficiently large feature space is low-rank. However, these nonlinear methods fail in the presence of sparse noise or outliers. In this work, we propose a new robust nonlinear factorization method called Robust Non-Linear Matrix Factorization (RNLMF). RNLMF const…
In the last couple of decades, there has been major advancements in the domain of missing data imputation. The techniques in the domain include amongst others: Expectation Maximization, Neural Networks with Evolutionary Algorithms or optimization techniques and K-Nearest Neighbor approaches to solve the problem. The presence of missing data entries in databases render the tasks of decision-making and data analysis nontrivial. As a result this area has attracted a lot of research interest with the aim being to yield accurate and time efficient and sensitive missing data imputation techniques es…
École Polytechnique, Institut Polytechnique de Paris（巴黎理工学院巴黎综合理工学院）；Digitalent lab (Moteur Intelligence Artificielle)（Digitalent实验室（人工智能引擎））
Missing values are a common problem in data science and machine learning. Removing instances with missing values can adversely affect the quality of further data analysis. This is exacerbated when there are relatively many more features than instances, and thus the proportion of affected instances is high. Such a scenario is common in many important domains, for example, single nucleotide polymorphism (SNP) datasets provide a large number of features over a genome for a relatively small number of individuals. To preserve as much information as possible prior to modeling, a rigorous imputation …
stMCDI: Masked Conditional Diffusion Model with Graph Neural Network for Spatial Transcriptomics Data Imputation
School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Hefei Institutes of Physical Science, Chinese Academy of Science（中国科学院合肥物质科学研究院）
University of Washington（华盛顿大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
In wearable sensing applications, data is inevitable to be irregularly sampled or partially missing, which pose challenges for any downstream application. An unique aspect of wearable data is that it is time-series data and each channel can be correlated to another one, such as x, y, z axis of accelerometer. We argue that traditional methods have rarely made use of both times-series dynamics of the data as well as the relatedness of the features from different sensors. We propose a model, termed as DynImp, to handle different time point's missingness with nearest neighbors along feature axis a…
Worcester Polytechnic Institute（伍斯特理工学院）；Colorado State University（科罗拉多州立大学）
In this paper, we demonstrate how deep autoencoders can be generalized to the case of inpainting and denoising, even when no clean training data is available. In particular, we show how neural networks can be trained to perform all of these tasks simultaneously. While, deep autoencoders implemented by way of neural networks have demonstrated potential for denoising and anomaly detection, standard autoencoders have the drawback that they require access to clean data for training. However, recent work in Robust Deep Autoencoders (RDAEs) shows how autoencoders can be trained to eliminate outliers…
TimeAutoDiff: A Unified Framework for Generation, Imputation, Forecasting, and Time-Varying Metadata Conditioning of Heterogeneous Time Series Tabular Data
Pretrained-Guided Conditional Diffusion Models for Microbiome Data Analysis
School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；College of Nursing Health Sciences, Yunnan Open University（云南开放大学护理与健康科学学院）
A Deep Latent Variable Framework for Jointly Modeling Missingness, Measurement Error, and Heterogeneity
University of Alberta（阿尔伯塔大学）；University of Western Ontario（西安大略大学）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；University of Prince Edward Island（爱德华王子岛大学）
