可扩展且分布式的轮廓近似
Scalable and Distributed Silhouette Approximation
AI总结:
提出基于采样的算法,以O(nkε^{-2}ln(nk/δ))次距离计算估计k-聚类的局部和全局轮廓,提供可证明的加性误差保证,并设计MapReduce和MPC框架下的分布式实现。
摘要:
轮廓是评估包含$n$个元素的数据集的$k$-聚类质量的最广泛使用的度量之一。其评估不需要聚类分配之外的任何信息。此外,轮廓非常易于解释,为衡量整个聚类或每个元素的质量提供分数。在一般度量下,精确计算数据集中每个元素的轮廓和聚类的全局轮廓需要$\Theta(n^2)$次距离计算。二次复杂度$\Theta(n^2)$极其昂贵,尤其是在大规模现代数据集上。令人惊讶的是,现有的使用$O(n^2)$次距离计算的近似方法是没有可证明和可控质量保证的启发式方法。我们引入了第一个严格且高效的算法来估计任意度量$k$-聚类的(局部)每个元素的轮廓和(全局)轮廓。我们的方法基于采样,执行$O(nk\varepsilon^{-2}\ln (nk/\delta))$次距离计算,并以至少$1-\delta$的概率提供加性误差$O(\varepsilon)$的估计。即,参数$\varepsilon$和$\delta$在$(0,1)$中控制精度与效率之间的权衡。我们还为MapReduce和大规模并行计算框架引入了可扩展且分布式的设计。我们的分布式算法使用常数轮次和次线性本地内存。最后,我们针对最先进的方法进行了大量实验。结果表明,我们的新技术在局部和全局轮廓估计中实现了精度与效率的最佳权衡。此外,我们的方法可以高效扩展到精确计算轮廓不可行的大规模数据集。
英文摘要:
The silhouette is one of the most widely used measures to assess the quality of a $k$-clustering of a dataset of $n$ elements. Its evaluation requires no information beyond the clustering assignment. In addition, the silhouette is extremely easy to interpret, providing a score to measure the quality of a clustering as a whole or for each element. The exact computation of the: (i) silhouette of each element of a dataset; and (ii) the global silhouette of the clustering; require $Θ(n^2)$ distance calculations, under general metrics. The quadratic complexity $Θ(n^2)$ is extremely prohibitive, especially on massive modern datasets. Surprisingly, existing approximate methods using $O(n^2)$ distance calculations are heuristics not offering provable and controllable guarantees on the quality of their results. We introduce the first rigorous and efficient algorithms to estimate: (i) the (local) silhouette of each element of a dataset; and (ii) the (global) silhouette; of any metric $k$-clustering. Our methods, based on sampling, perform $O(nk\varepsilon^{-2}\ln (nk/δ))$ distance computations, and provide estimates with additive error $O(\varepsilon)$ with probability at least $1-δ$. That is, parameters $\varepsilon$ and $δ$ in $(0,1)$ control the trade-off between accuracy and efficiency. We also introduce a scalable and distributed design of our methods for the MapReduce and Massively Parallel Computing (MPC) frameworks. Our distributed algorithms use a constant number of rounds and sublinear local memory. Finally, we perform extensive experiments against state-of-the-art approaches. The results show that our new techniques yield the best trade-off between accuracy and efficiency for both local and global silhouette estimation. In addition, our methods scale efficiently to massive datasets for which an exact computation of the silhouette is not practical.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。