并行聚类算法中高维数据的高效划分方法
An Efficient Method of Partitioning High Volumes of Multidimensional Data for Parallel Clustering Algorithms
摘要：
最优数据划分是并行和分布式实现聚类算法的必要计算，因为它确保独立的任务完成，公平分布，较少的受影响点的数量和更好和更快的合并。虽然使用Kd树的分区在学术界被常规地使用，但是随着数据维度的增加，它遭受性能下降和偏差（非均匀分布），因此不适合在维度可以是100到1000的数量级的工业中实际使用。为了解决这些问题，我们提出了两个新的分区技术，使用现有的数学模型和研究其可行性，性能（偏见和分区速度）和可能的变化，在选择初始种子。第一种方法使用基于n维散列网格的方法，该方法基于将空间中的点映射到散列点的一组立方体。第二种方法使用voronoi平面树，其中每个平面对应于一个分区。我们发现基于网格的方法在计算上是不切实际的，而使用voronoi平面的树（使用可扩展的K-Means++初始种子）随着维度的增加而大大优于Kd树方法。
英文摘要：
An optimal data partitioning in parallel & distributed implementation of clustering algorithms is a necessary computation as it ensures independent task completion, fair distribution, less number of affected points and better & faster merging. Though partitioning using Kd Tree is being conventionally used in academia, it suffers from performance drenches and bias (non equal distribution) as dimensionality of data increases and hence is not suitable for practical use in industry where dimensionality can be of order of 100s to 1000s. To address these issues we propose two new partitioning techniques using existing mathematical models & study their feasibility, performance (bias and partitioning speed) & possible variants in choosing initial seeds. First method uses an n dimensional hashed grid based approach which is based on mapping the points in space to a set of cubes which hashes the points. Second method uses a tree of voronoi planes where each plane corresponds to a partition. We found that grid based approach was computationally impractical, while using a tree of voronoi planes (using scalable K-Means++ initial seeds) drastically outperformed the Kd-tree tree method as dimensionality increased.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
