大数据中的大漏洞:一种检测高维数据中大超矩形的Monte Carlo算法
Big Holes in Big Data: A Monte Carlo Algorithm for Detecting Large Hyper-rectangles in High Dimensional Data
摘要:
我们提出了第一个算法,用于发现洞在高维数据中运行在多项式时间的维数。以前的算法是指数的。在2D和3D空间中的一组点中找到大的空矩形或框已经得到了很好的研究。存在有效的算法来识别这些低维空间中的空区域。不幸的是,这样的效率是缺乏在更高的维度,其中的问题已被证明是NP完全的,当尺寸包括在输入中。发现大空白空间的算法的应用包括大数据分析、推荐系统、自动知识发现和查询优化。我们的蒙特卡洛算法发现有趣的最大空超矩形的情况下,维数和输入大小,否则分析不切实际。运行时在输入的大小和维数方面是多项式的。我们将该算法应用于蛋白质结构的39维数据集,并发现了我们认为无法以其他方式推断的有趣特性。
英文摘要:
We present the first algorithm for finding holes in high dimensional data that runs in polynomial time with respect to the number of dimensions. Previous algorithms are exponential. Finding large empty rectangles or boxes in a set of points in 2D and 3D space has been well studied. Efficient algorithms exist to identify the empty regions in these low-dimensional spaces. Unfortunately such efficiency is lacking in higher dimensions where the problem has been shown to be NP-complete when the dimensions are included in the input. Applications for algorithms that find large empty spaces include big data analysis, recommender systems, automated knowledge discovery, and query optimization. Our Monte Carlo-based algorithm discovers interesting maximal empty hyper-rectangles in cases where dimensionality and input size would otherwise make analysis impractical. The run-time is polynomial in the size of the input and the number of dimensions. We apply the algorithm on a 39-dimensional data set for protein structures and discover interesting properties that we think could not be inferred otherwise.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。