基于数据划分的高效大规模聚类
Efficient Large Scale Clustering based on Data Partitioning
摘要:
聚类技术在数据集模式的提取和识别方面具有很强的吸引力。然而,他们的应用程序非常大的空间数据集提出了许多挑战,如高维数据,异构性和一些算法的高复杂性。例如,一些算法可能具有线性复杂度,但它们需要领域知识来确定它们的输入参数。分布式集群技术构成了大数据挑战的一个非常好的替代方案(例如,Volume、Variety、Veracity和Velocity)。这些技术通常包括两个阶段。第一阶段生成局部模型或模式,第二阶段倾向于聚合局部结果以获得全局模型。虽然第一阶段可以在每个站点上并行执行,因此是高效的,但聚合阶段是复杂的、耗时的,并且可能产生不正确和模糊的全局聚类,从而产生不正确的模型。在本文中,我们提出了一种新的分布式聚类方法,有效地处理这两个阶段,生成的本地结果和生成的全球模型的聚合。对于第一阶段,我们的方法是能够分析位于每个网站使用不同的聚类技术的数据集。聚合阶段的设计方式是,最终的集群是紧凑和准确的,而整个过程是有效的时间和内存分配。对于评估,我们使用两个著名的聚类算法,K-Means和DBSCAN。这种分布式聚类技术的一个关键输出是全局聚类的数量是动态的,不需要预先固定。实验结果表明,该方法具有可扩展性,并产生高质量的结果。
英文摘要:
Clustering techniques are very attractive for extracting and identifying patterns in datasets. However, their application to very large spatial datasets presents numerous challenges such as high-dimensionality data, heterogeneity, and high complexity of some algorithms. For instance, some algorithms may have linear complexity but they require the domain knowledge in order to determine their input parameters. Distributed clustering techniques constitute a very good alternative to the big data challenges (e.g.,Volume, Variety, Veracity, and Velocity). Usually these techniques consist of two phases. The first phase generates local models or patterns and the second one tends to aggregate the local results to obtain global models. While the first phase can be executed in parallel on each site and, therefore, efficient, the aggregation phase is complex, time consuming and may produce incorrect and ambiguous global clusters and therefore incorrect models. In this paper we propose a new distributed clustering approach to deal efficiently with both phases, generation of local results and generation of global models by aggregation. For the first phase, our approach is capable of analysing the datasets located in each site using different clustering techniques. The aggregation phase is designed in such a way that the final clusters are compact and accurate while the overall process is efficient in time and memory allocation. For the evaluation, we use two well-known clustering algorithms, K-Means and DBSCAN. One of the key outputs of this distributed clustering technique is that the number of global clusters is dynamic, no need to be fixed in advance. Experimental results show that the approach is scalable and produces high quality results.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。