二进制数据的维数是多少?
What is the dimension of your binary data?
摘要:
许多0/1数据集有非常多的变量;另一方面,它们是稀疏的,变量的依赖结构比变量的数量更简单。定义这样一个数据集的有效维数是一个重要的问题。我们考虑的问题,定义一个强大的0/1数据集的维数测量,并表明,分形维数的基本思想可以适用于二进制数据。然而,这样的分形维数是难以解释的。为此,我们引入了归一化分维的概念。对于数据集$D$,其归一化分形维数是数据集$D'$中具有独立列且具有与$D$相同(未归一化)分形维数的列数。归一化分形维数度量数据的依赖结构的程度。研究了归一化分维的性质,讨论了归一化分维的计算方法。我们给出了归一化分形维数的实证结果,将其与PCA等基线措施进行比较。研究了整个数据集的维数与聚类形成的子组维数之间的关系。结果表明数据集之间和数据集内存在有趣的差异。
英文摘要:
Many 0/1 datasets have a very large number of variables; on the other hand, they are sparse and the dependency structure of the variables is simpler than the number of variables would suggest. Defining the effective dimensionality of such a dataset is a nontrivial problem. We consider the problem of defining a robust measure of dimension for 0/1 datasets, and show that the basic idea of fractal dimension can be adapted for binary data. However, as such the fractal dimension is difficult to interpret. Hence we introduce the concept of normalized fractal dimension. For a dataset $D$, its normalized fractal dimension is the number of columns in a dataset $D'$ with independent columns and having the same (unnormalized) fractal dimension as $D$. The normalized fractal dimension measures the degree of dependency structure of the data. We study the properties of the normalized fractal dimension and discuss its computation. We give empirical results on the normalized fractal dimension, comparing it against baseline measures such as PCA. We also study the relationship of the dimension of the whole dataset and the dimensions of subgroups formed by clustering. The results indicate interesting differences between and within datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。