arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1601.07251stat.ME
收藏

检测偏离的数据单元

Detecting deviating data cells

Peter J. Rousseeuw, Wannes Van den Bossche

摘要:

多变量数据集由$d$维度中的$n$案例组成,并且通常存储在$n$乘$d$数据矩阵中。众所周知,真实数据可能包含离群值。根据具体情况,离群值可能是(a)可能对数据分析产生不利影响的不良错误,或(b)意外信息的宝贵金块。在统计学和数据分析中,离群值这个词通常指的是数据矩阵中的一行,检测这种离群值的方法只有在至少一半的行是干净的时候才有效。但通常许多行都有一些受污染的单元格值,单独查看每个变量(列)可能无法看到这些值。我们提出了第一种方法来检测偏离数据单元在一个多元样本,考虑到变量之间的相关性。它对干净行的数量没有限制,并且可以处理高维。其他优点是,它提供了外围单元格的“预期”值的估计数,同时估算了缺失值。我们说明了几个真实的数据集上的方法,它揭示了更多的结构比发现纯粹的列方法或纯粹的rowwise方法。所提出的方法可以帮助诊断为什么某一行是离群的,例如在过程控制中。它也可以作为估计多变量位置和散布矩阵的初始步骤。

英文摘要:

A multivariate dataset consists of $n$ cases in $d$ dimensions, and is often stored in an $n$ by $d$ data matrix. It is well-known that real data may contain outliers. Depending on the situation, outliers may be (a) undesirable errors which can adversely affect the data analysis, or (b) valuable nuggets of unexpected information. In statistics and data analysis the word outlier usually refers to a row of the data matrix, and the methods to detect such outliers only work when at least half the rows are clean. But often many rows have a few contaminated cell values, which may not be visible by looking at each variable (column) separately. We propose the first method to detect deviating data cells in a multivariate sample which takes the correlations between the variables into account. It has no restriction on the number of clean rows, and can deal with high dimensions. Other advantages are that it provides estimates of the `expected' values of the outlying cells, while imputing missing values at the same time. We illustrate the method on several real data sets, where it uncovers more structure than found by purely columnwise methods or purely rowwise methods. The proposed method can help to diagnose why a certain row is outlying, e.g. in process control. It may also serve as an initial step for estimating multivariate location and scatter matrices.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑