现实世界数据集中缺失值的普遍存在对统计推断构成了挑战,并可能阻碍相似数据集在同一研究中被分析,从而排除了许多现有数据集用于新分析的可能性。尽管已开发了大量用于数据插补的包和算法,但如果存在大量缺失值和低样本量,绝大多数算法表现不佳,而这些不幸是实证数据中的常见特征。这种低准确度的估计会不利地影响下游统计模型的性能。我们开发了一个在存在缺失数据且无需插补的情况下进行回归和分类的统计推断框架。我们的框架RIFLE(通过低阶矩估计进行稳健推断,Robust InFerence via Low-order moment Estimations)估计底层数据分布的低阶矩及相应的置信区间,以学习一个分布稳健的模型。我们将该框架专门应用于线性回归和正态判别分析,并提供了收敛性和性能保证。该框架也可适配用于插补缺失数据。在数值实验中,我们在存在缺失值的情况下,将RIFLE与几种最先进的方法(包括MICE、Amelia、MissForest、KNN-imputer、MIDA和Mean Imputer)在插补和推断方面进行了比较。我们的实验表明,当缺失值百分比较高和/或数据点数量相对较少时,RIFLE优于其他基准算法。RIFLE已在 https://github.com/optimization-for-data-driven-science/RIFLE 上公开提供。
英文摘要:
The ubiquity of missing values in real-world datasets poses a challenge for statistical inference and can prevent similar datasets from being analyzed in the same study, precluding many existing datasets from being used for new analyses. While an extensive collection of packages and algorithms have been developed for data imputation, the overwhelming majority perform poorly if there are many missing values and low sample sizes, which are unfortunately common characteristics in empirical data. Such low-accuracy estimations adversely affect the performance of downstream statistical models. We develop a statistical inference framework for regression and classification in the presence of missing data without imputation. Our framework, RIFLE (Robust InFerence via Low-order moment Estimations), estimates low-order moments of the underlying data distribution with corresponding confidence intervals to learn a distributionally robust model. We specialize our framework to linear regression and normal discriminant analysis, and we provide convergence and performance guarantees. This framework can also be adapted to impute missing data. In numerical experiments, we compare RIFLE to several state-of-the-art approaches (including MICE, Amelia, MissForest, KNN-imputer, MIDA, and Mean Imputer) for imputation and inference in the presence of missing values. Our experiments demonstrate that RIFLE outperforms other benchmark algorithms when the percentage of missing values is high and/or when the number of data points is relatively small. RIFLE is publicly available at https://github.com/optimization-for-data-driven-science/RIFLE.
Distributionally Faithful Imputation via Positive Semi-Definite Kernel Density Estimation
Inria - Ecole Normale Supériore Paris(法国巴黎国家信息与自动化研究所-巴黎高等师范学院);Department of Computer Science - UCL(伦敦大学学院计算机科学系);SDA Bocconi School of Management(博科尼管理学院)
Marginalize, Rather than Impute: Probabilistic Wind Power Forecasting with Incomplete Data
School of Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电气工程学院);Dyson School of Design Engineering, Imperial College London(帝国理工学院设计工程学院);Department of Technology, Management and Economics, Technical University of Denmark(丹麦技术大学技术、管理与经济系);Centre for Energy Research – CoRE, Aarhus University(奥胡斯大学能源研究中心)
Handling Overlapping Asymmetric Datasets -- A Twice Penalized P-Spline Approach
Newcastle University(纽卡斯尔大学);School of Computing(计算机学院);School of Mathematics, Statistics and Physics(数学、统计与物理学院);Translational & Clinical Research Institute(转化与临床研究院)
Synthetic data for ratemaking: imputation-based methods vs adversarial networks and autoencoders
Department of Actuarial Science, Faculty of Business and Economics, University of Lausanne(卢塞恩大学商学院精算科学系);Institute of Mathematics and Statistics, Faculty of Science and Technology, University of Tartu(塔尔图大学科学与技术学院数学与统计学研究所)
TabPFN Extensions for Interpretable Geotechnical Modelling
Department of Civil and Environmental Engineering, Tohoku University(东北大学土木环境工程系);Research Organization of Information and Systems, The Institute of Statistical Mathematics(信息与系统研究所统计数学研究所);Department of Statistical Science, The Graduate University for Advanced Studies(高级研究大学统计科学系)
Electricité De France R&D(法国电力公司研发部);INRIA(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);École polytechnique(巴黎综合理工学院);Institut Polytechnique de Paris(巴黎理工学院);Technion - Israel Institute of Technology(以色列理工学院)