一种新的K-means聚类方法
A New Parallelization Method for K-means
摘要：
K-means是数据采掘领域中一种常用的聚类方法。为了处理大型数据集，研究人员提出了PKMeans，这是MapReduce上的并行k-means。然而，现有的k-means并行化方法，包括PKMeans有很多局限性。PKMeans无法在一个MapReduce作业中完成所有迭代，因此它必须在循环中重复级联MapReduce作业，直到收敛。在最流行的MapReduce平台Hadoop上，每个MapReduce作业都会在作业启动和洗牌阶段引入大量的I/O开销和额外的执行时间。更糟糕的是，已经证明，在最坏的情况下，k-means需要$2^{Ω(n)}$ MapReduce作业来收敛，其中n是数据实例的数量，这意味着对于大型数据集来说开销巨大。此外，在PKMeans中，最多可以为每个质心分配一个reducer并更新每个质心，因此PKMeans只能使用有限数量的并行reducer。本文提出了一种改进的k-means并行方法IPKMeans，它具有一个使用k-d树的并行预处理阶段，可以在一个MapReduce作业中完成k-means，与PKMeans相比，它具有更多的reducer并行工作和更低的I/O开销，并且具有快速的后处理阶段生成最终结果。在我们的方法中，k-d树和新的改进的并行k-means使用MapReduce实现，并在Hadoop上进行测试。实验结果表明，在相同的数据集和初始质心下，该方法比PKMeans方法具有2/3的低I/O开销和更少的时间来获得非常接近的聚类结果。
英文摘要：
K-means is a popular clustering method used in data mining area. To work with large datasets, researchers propose PKMeans, which is a parallel k-means on MapReduce. However, the existing k-means parallelization methods including PKMeans have many limitations. PKMeans can't finish all its iterations in one MapReduce job, so it has to repeat cascading MapReduce jobs in a loop until convergence. On the most popular MapReduce platform, Hadoop, every MapReduce job introduces significant I/O overheads and extra execution time at stages of job start-up and shuffling. Even worse, it has been proved that in the worst case, k-means needs $2^{Ω(n)}$ MapReduce jobs to converge, where n is the number of data instances, which means huge overheads for large datasets. Additionally, in PKMeans, at most one reducer can be assigned to and update each centroid, so PKMeans can only make use of limited number of parallel reducers. In this paper, we propose an improved parallel method for k-means, IPKMeans, which has a parallel preprocessing stage using k-d tree and can finish k-means in one single MapReduce job with much more reducers working in parallel and lower I/O overheads than PKMeans and has a fast post-processing stage generating the final result. In our method, both k-d tree and the new improved parallel k-means are implemented using MapReduce and tested on Hadoop. Our experiments show that with same dataset and initial centroids, our method has up to 2/3 lower I/O overheads and consumes less amount of time than PKMeans to get a very close clustering result.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
