arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.00752math.STstat.TH
收藏

存在缺失值与逐单元污染的鲁棒协方差估计

Robust covariance estimation with missing values and cell-wise contamination

Karim Lounici, Grégoire Pacreau

AI总结:

针对含缺失值和逐单元污染的数据集协方差估计难题,提出无需插补的无偏协方差估计量,结合异常值检测处理高维低秩场景污染,理论与实验均验证其优于现有方法。

摘要:

大型数据集常受缺失或错误数据形式的逐单元异常值影响。然而,丢弃任何含异常值的样本可能导致数据集过小,无法准确估计协方差矩阵。此外,为解决该问题设计的鲁棒方法需要协方差算子可逆,因此在高维数据上效果不佳。本文中,我们提出一种存在缺失值时的无偏协方差估计量,它无需任何插补步骤,仍能在算子范数下达到接近极小极大的统计精度。我们还提倡将其与逐单元异常值检测方法结合使用,以处理高维低秩场景下的逐单元污染——现有最优方法在该场景下可能存在数值不稳定和计算时间长的问题。为补充我们的理论发现,我们开展了一项实验研究,证明了我们的方法在低维和高维场景下均优于现有最优方法。

英文摘要:

Large datasets are often affected by cell-wise outliers in the form of missing or erroneous data. However, discarding any samples containing outliers may result in a dataset that is too small to accurately estimate the covariance matrix. Moreover, the robust procedures designed to address this problem require the invertibility of the covariance operator and thus are not effective on high-dimensional data. In this paper, we propose an unbiased estimator for the covariance in the presence of missing values that does not require any imputation step and still achieves near minimax statistical accuracy with the operator norm. We also advocate for its use in combination with cell-wise outlier detection methods to tackle cell-wise contamination in a high-dimensional and low-rank setting, where state-of-the-art methods may suffer from numerical instability and long computation times. To complement our theoretical findings, we conducted an experimental study which demonstrates the superiority of our approach over the state of the art both in low and high dimension settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑