Clustering is a fundamental tool in statistical machine learning in the presence of heterogeneous data. Most recent results focus primarily on optimal mislabeling guarantees when data are distributed around centroids with sub-Gaussian errors. Yet, the restrictive sub-Gaussian model is often invalid in practice since various real-world applications exhibit heavy tail distributions around the centroids or suffer from possible adversarial attacks that call for robust clustering with a robust data-driven initialization. In this paper, we present initialization and subsequent clustering methods that provably guarantee near-optimal mislabeling for general mixture models when the number of clusters and data dimensions are finite. We first introduce a hybrid clustering technique with a novel multivariate trimmed mean type centroid estimate to produce mislabeling guarantees under a weak initialization condition for general error distributions around the centroids. A matching lower bound is derived, up to factors depending on the number of clusters. In addition, our approach also produces similar mislabeling guarantees even in the presence of adversarial outliers. Our results reduce to the sub-Gaussian case in finite dimensions when errors follow sub-Gaussian distributions. To solve the problem thoroughly, we also present novel data-driven robust initialization techniques and show that, with probabilities approaching one, these initial centroid estimates are sufficiently good for the subsequent clustering algorithm to achieve the optimal mislabeling rates. Furthermore, we demonstrate that the Lloyd algorithm is suboptimal for more than two clusters even when errors are Gaussian and for two clusters when error distributions have heavy tails. Both simulated data and real data examples further support our robust initialization procedure and clustering algorithm.
Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection
Delft Center for Systems and Control (DCSC), Delft University of Technology, The Netherlands(代尔夫特系统与控制中心(DCSC),代尔夫特理工大学,荷兰);School of Computer Science and Statistics, Trinity College Dublin, Ireland(三一学院都柏林计算机科学与统计学系,爱尔兰);Departments of Aerospace Engineering Sciences and Computer Science, University of Colorado Boulder, USA(科罗拉多大学波德分校航空航天工程科学系与计算机科学系,美国);The Italian Institute of Artificial Intelligence (AI4I), Italy(意大利人工智能研究所(AI4I),意大利)
Efficient and Accurate Learning of Mixtures of Plackett-Luce Models
University of Pennsylvania(宾夕法尼亚大学)
Plackett-Luce(PL)混合模型是最基本的排序模型之一,是一个具有重要理论和实际意义的研究热点。大多数先前提出的参数估计算法实例化的EM算法,往往与随机初始化。然而,这样的初始化方案可能不会产生良好的初始估计,并且算法需要多次重启,从而导致大的时间复杂度。至于EM过程,虽然可以有效地执行E步骤,但由于PL似然函数的组合性质,很难最大化M步骤中的对数似然(Gormley和Murphy 2008)。因此,以前的作者倾向于最大化替代似然函数的算法(Zhao et al. 2018,2020)。然而,最终的估计可能会偏离真正的最大似然估计的结果。在本文中,我们解决这些已知的限制。我们提出了一个初始化算法,可以提供一个可证明准确的初始估计和EM算法,有效地最大化真正的对数似然函数。在合成数据集和真实数据集上的实验表明,该算法在准确性和速度方面与基线算法具有竞争力,特别是在具有大量项目的数据集上。
The EM-algorithm and the Method of Moments in Softmax Mixture Models
Department of Statistical Sciences, University of Toronto(多伦多大学统计学系);Department of Statistics and Data Science, Cornell University(康奈尔大学统计学与数据科学系);Courant Institute of Mathematical Sciences and Center for Data Science, New York University(纽约大学Courant数学科学研究所和数据科学中心);Department of Mathematics and Department of Statistics and Data Science, Cornell University(康奈尔大学数学系和统计学与数据科学系)
Machine Learning Department(机器学习系);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫比内扎伊德人工智能大学);Remote Sensing Department(遥感系);National Center of Space Research and Technology(国家航天研究与技术中心);Department of Mathematical Sciences(数学科学系);Purdue University Fort Wayne(普渡大学富尔顿分校);Department of Mathematics and Statistics(数学与统计学系);University of Minnesota Duluth(明尼苏达大学杜鲁斯分校);Department of Mathematics(数学系);Nazarbayev University(纳扎尔巴耶夫大学)
Universality of high-dimensional scaling limits of stochastic gradient descent
Department of Mathematics, Northwestern University(西北大学数学系);Department of Statistics and Actuarial Science, Department of Applied Mathematics, and Cheriton School of Computer Science, University of Waterloo(滑铁卢大学统计与精算科学系、应用数学系和切尔顿计算机科学学院)
Consistency of Lloyd's Algorithm Under Perturbations
Department of Mathematics and Statistics, McGill University(数学与统计学系,麦吉尔大学);Department of Statistics and Operations Research, University of North Carolina at Chapel Hill(统计与运筹学系,北卡罗来纳大学教堂山分校);Department of Statistics, University of Michigan(统计学系,密歇根大学)