The EM-algorithm and the Method of Moments in Softmax Mixture Models
Department of Statistical Sciences, University of Toronto(多伦多大学统计学系);Department of Statistics and Data Science, Cornell University(康奈尔大学统计学与数据科学系);Courant Institute of Mathematical Sciences and Center for Data Science, New York University(纽约大学Courant数学科学研究所和数据科学中心);Department of Mathematics and Department of Statistics and Data Science, Cornell University(康奈尔大学数学系和统计学与数据科学系)
Xin Bing, Florentina Bunea, Jonathan Niles-Weed, Marten Wegkamp
Softmax Mixture Models (SMMs) are discrete $K$-component mixture models for the probabilities of selecting one of $p$ candidate feature vectors $X_1,\ldots,X_p\in\mathbb{R}^L$ in heterogeneous populations and are widely used in econometrics and scientific applications. Related softmax mixture mechanisms also appear in modern LLM architectures. We provide a theoretical and methodological study of SMMs, focusing on the Expectation-Maximization (EM) algorithm and the Method of Moments (MoM). We show that EM recovers the mixture atoms at the parametric rate, up to logarithmic factors, after $\mathcal{O}(\log N)$ iterations, provided atom separation is at least of order $\log K$. This improves on separation conditions in existing analyses of EM for high-dimensional Gaussian mixtures. We also develop MoM procedures for parameter and subspace estimation. Although MoM parameter estimates converge more slowly than EM and can deteriorate with $K$, they provide provable warm starts for EM and are useful for small $K$. For general $K$, we estimate the atom subspace via MoM and recommend running EM from multiple random initializations within this subspace. Finally, as $p\to\infty$, we show that SMMs approximate mixtures of exponential tilts of the feature distribution, yielding asymptotic identifiability.
Demystifying Softmax Gating Function in Gaussian Mixture of Experts
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Univ. Grenoble Alpes, Inria(格勒诺布尔阿尔卑斯大学,法国国家信息与自动化研究所);CNRS, Grenoble INP, LJK(法国国家科学研究中心,格勒诺布尔理工学院,让·库尔研究所)
Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models
School of Computing, National University of Singapore, Singapore.(新加坡国立大学计算机学院);ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems(细胞系统数学分析卓越研究中心);School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia.(昆士兰科技大学数学科学学院);Department of Statistics and Data Science, University of Texas at Austin, Austin, USA(德克萨斯大学奥斯汀分校统计与数据科学系)
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Univ. Grenoble Alpes(格勒诺布尔大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔理工学院);LJK(约瑟夫·傅里叶实验室);The University of Queensland(昆士兰大学)
On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
Department of Computer Science(计算机科学系);Department of Statistics and Data Sciences(统计学与数据科学系);Department of Electrical and Computer Engineering(电气与计算机工程系)
Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam(越南胡志明市科学大学数学与计算机科学学院);Vietnam National University Ho Chi Minh City, Vietnam(越南胡志明市国家大学);Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam(越南胡志明市科学大学信息技术学院);ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems(细胞系统数学分析 excellence 中心);School of Mathematical Sciences, Queensland University of Technology, Brisbane City, Australia(昆士兰科技大学数学科学学院);Department of Statistics and Data Science, University of Texas at Austin, Austin, USA(德克萨斯大学奥斯汀分校统计与数据科学系)
Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average
School of Computer Science and Informatics, Cardiff University(卡迪夫大学计算机科学与信息学学院);Department of Computer Engineering, Turkish Air Force Academy, National Defence University(土耳其空军学院计算机工程系,国家防卫大学)
Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts
School of Computing, National University of Singapore(新加坡国立大学计算机学院);ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems(细胞系统数学分析卓越中心);School of Mathematical Sciences, Queensland University of Technology(昆士兰科技大学数学科学学院);Laboratoire d’Analyse et d’Architecture des Systèmes, CNRS(系统分析与架构实验室,CNRS);School of Medicine and Dentistry, Griffith University(格里菲斯大学医学与牙科学院);Department of Mathematics and Physical Sciences, La Trobe University(拉特罗布大学数学与物理科学系);Institute of Mathematics for Industry, Kyushu University(工业数学研究所,九州大学);Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City(胡志明市大学数学与计算机科学学院);Vietnam National University, Ho Chi Minh City(越南国家大学,胡志明市);Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK(格勒诺布尔阿尔卑斯大学,Inria,CNRS,格勒诺布尔INP,LJK)
Density estimation via mixture discrepancy and moments
School of Mathematical Sciences, Peking University, Beijing 100871, China(北京大学数学科学学院);School of Mathematical Sciences, Beijing Normal University, Beijing 100875, China(北京师范大学数学科学学院)
Nonlinearity and Uncertainty Informed Moment-Matching Gaussian Mixture Splitting
Department of Mechanical and Aerospace Engineering, Utah State University(机械与航空航天工程系,犹他州立大学);School of Aeronautics and Astronautics, Purdue University(航空航天工程学院,普渡大学)