arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09903stat.MLcs.LGmath.STstat.MEstat.TH
收藏

EM算法与矩方法在Softmax混合模型中的应用

The EM-algorithm and the Method of Moments in Softmax Mixture Models

Department of Statistical Sciences, University of Toronto(多伦多大学统计学系);Department of Statistics and Data Science, Cornell University(康奈尔大学统计学与数据科学系);Courant Institute of Mathematical Sciences and Center for Data Science, New York University(纽约大学Courant数学科学研究所和数据科学中心);Department of Mathematics and Department of Statistics and Data Science, Cornell University(康奈尔大学数学系和统计学与数据科学系)

Xin Bing, Florentina Bunea, Jonathan Niles-Weed, Marten Wegkamp

AI总结:

本文研究Softmax混合模型,证明EM算法在温和分离条件下以参数速率收敛,并开发矩方法用于参数估计和子空间估计,为EM提供冷启动,同时证明模型在大p下渐近可识别。

摘要:

Softmax混合模型(SMMs)是离散的$K$分量混合模型,用于描述异质群体中选择$p$个候选特征向量$X_1,\ldots,X_p\in\mathbb{R}^L$之一的概率,广泛应用于计量经济学和科学应用中。相关的softmax混合机制也出现在现代LLM架构中。我们对SMMs进行了理论和方法论研究,重点关注期望最大化(EM)算法和矩方法(MoM)。我们证明,在原子分离度至少为$\log K$量级的条件下,EM算法经过$\mathcal{O}(\log N)$次迭代后,能以参数速率(至多相差对数因子)恢复混合原子。这改进了现有高维高斯混合模型EM分析中的分离条件。我们还开发了用于参数和子空间估计的MoM程序。尽管MoM参数估计的收敛速度慢于EM,且可能随$K$增大而恶化,但它们为EM提供了可证明的冷启动(warm starts),并且对于较小的$K$很有用。对于一般的$K$,我们通过MoM估计原子子空间,并建议在该子空间内从多个随机初始化运行EM。最后,当$p\to\infty$时,我们证明SMMs近似于特征分布的指数倾斜混合,从而产生渐近可识别性。

英文摘要:

Softmax Mixture Models (SMMs) are discrete $K$-component mixture models for the probabilities of selecting one of $p$ candidate feature vectors $X_1,\ldots,X_p\in\mathbb{R}^L$ in heterogeneous populations and are widely used in econometrics and scientific applications. Related softmax mixture mechanisms also appear in modern LLM architectures. We provide a theoretical and methodological study of SMMs, focusing on the Expectation-Maximization (EM) algorithm and the Method of Moments (MoM). We show that EM recovers the mixture atoms at the parametric rate, up to logarithmic factors, after $\mathcal{O}(\log N)$ iterations, provided atom separation is at least of order $\log K$. This improves on separation conditions in existing analyses of EM for high-dimensional Gaussian mixtures. We also develop MoM procedures for parameter and subspace estimation. Although MoM parameter estimates converge more slowly than EM and can deteriorate with $K$, they provide provable warm starts for EM and are useful for small $K$. For general $K$, we estimate the atom subspace via MoM and recommend running EM from multiple random initializations within this subspace. Finally, as $p\to\infty$, we show that SMMs approximate mixtures of exponential tilts of the feature distribution, yielding asymptotic identifiability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑