arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.05786cs.LG
收藏

用于处理异常值与重尾分布的双层直方图

Two-level histograms for dealing with outliers and heavy tail distributions

Orange Labs(奥兰治实验室)

Marc Boullé

AI总结:

针对G-Enum直方图在异常值与重尾分布下的局限,提出基于对数变换划分子集、再构建聚合子直方图的双层启发式方法,实验验证了其有效性。

摘要:

直方图是探索性分析中用于总结单变量分布的最常用方法之一。其中,不规则直方图是优秀的非参数密度估计器,仅需极少参数:分箱的数量及其长度、频率。现有文献已提出诸多推断这些参数的方法,要么对底层数据分布做出假设,要么采用模型选择方法。本文聚焦于G-Enum直方图方法,该方法利用最小描述长度(Minimum Description Length,MDL)准则构建直方图,无需任何用户自定义参数,在准确率、简洁性和计算时间方面均达到当前最优性能。我们探究了该方法在存在异常值或重尾分布场景下的局限性,提出了一种双层启发式方法来应对这类情况。第一层利用数据的对数变换,将数据集划分为一组数值范围可控的数据子集;第二层为每个数据子集构建子直方图,再将这些子直方图聚合得到完整的直方图。大量实验验证了该方法的有效性。

英文摘要:

Histograms are among the most popular methods used in exploratory analysis to summarize univariate distributions. In particular, irregular histograms are good non-parametric density estimators that require very few parameters: the number of bins with their lengths and frequencies. Many approaches have been proposed in the literature to infer these parameters, either assuming hypotheses about the underlying data distributions or exploiting a model selection approach. In this paper, we focus on the G-Enum histogram method, which exploits the Minimum Description Length (MDL) principle to build histograms without any user parameter and achieves state-of-the art performance w.r.t accuracy; parsimony and computation time. We investigate on the limits of this method in the case of outliers or heavy-tailed distributions. We suggest a two-level heuristic to deal with such cases. The first level exploits a logarithmic transformation of the data to split the data set into a list of data subsets with a controlled range of values. The second level builds a sub-histogram for each data subset and aggregates them to obtain a complete histogram. Extensive experiments show the benefits of the approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑