Convergence Rates for Softmax Gating Mixture of Experts
The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
备注：
Section 2 of this work comes from our previous paper titled "On Least Square Estimation in Softmax Gating Mixture of Experts" and published at the ICML 2024
Mixture of experts (MoE) has recently emerged as an effective framework to advance the efficiency and scalability of machine learning models by softly dividing complex tasks among multiple specialized sub-models termed experts. Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights. Despite its widespread use in practice, a comprehensive study on the effects of the softmax gating on the MoE has been lacking in the literature. To bridge this gap in this paper, we perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating, respectively. Furthermore, our theories also provide useful insights into the design of sample-efficient expert structures. In particular, we demonstrate that it requires polynomially many data points to estimate experts satisfying our proposed \emph{strong identifiability} condition, namely a commonly used two-layer feed-forward network. In stark contrast, estimating linear experts, which violate the strong identifiability condition, necessitates exponentially many data points as a result of intrinsic parameter interactions expressed in the language of partial differential equations. All the theoretical results are substantiated with a rigorous guarantee.
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（约瑟夫·傅里叶实验室）；The University of Queensland（昆士兰大学）
Demystifying Softmax Gating Function in Gaussian Mixture of Experts
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Univ. Grenoble Alpes, Inria（格勒诺布尔阿尔卑斯大学，法国国家信息与自动化研究所）；CNRS, Grenoble INP, LJK（法国国家科学研究中心，格勒诺布尔理工学院，让·库尔研究所）
Softmax是一种输出激活函数，用于在许多深度学习应用中对分类概率分布进行建模。然而，最近的一项研究表明，softmax可能是语言建模中神经网络表示能力的瓶颈（softmax瓶颈）。在本文中，我们提出了一个输出激活函数，用于在没有额外参数的情况下打破softmax瓶颈。我们从log-softmax的输出集的角度重新分析了softmax瓶颈，并确定了softmax瓶颈的原因。在此分析的基础上，我们提出了sigsoftmax，它是由指数函数和sigmoid函数的乘法。Sigsoftmax可以打破softmax的瓶颈。语言建模实验表明，sigsoftmax和mixed of sigsoftmax分别优于softmax和mixed of softmax。
On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
Department of Computer Science（计算机科学系）；Department of Statistics and Data Sciences（统计学与数据科学系）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Mixture of ELM based experts with trainable gating network
Yazd University（亚兹德大学）
混合专家模型方法是一种基于神经网络的集成学习方法，具有很强的提高整体分类精度的能力。该方法基于分而治之的原则，通过选通网络的监督将问题空间划分给多个专家。本文提出了一种基于混合专家模型的集成学习方法--MEETG（mixed of ELM based experts with trainable gating networks），它由多层感知器（MLP）和门网络组成，其中应用基于梯度的学习算法来训练MLP，这是一个迭代且耗时的过程。为了克服这些问题，我们利用极限学习机（ELM）的优点设计了ME的结构，ELM作为单隐层前馈神经网络的学习算法，与其他传统的学习算法相比，具有更快的学习速度和更好的泛化能力。此外，在所提出的方法中，一个可训练的门控网络被应用于根据输入样本动态地聚集专家的输出。在11个基准数据集上的实验结果和统计分析证实了MEETG在分类问题上具有可接受的性能。此外，我们的实验结果表明，该方法在预测稳定性和分类准确性方面优于原始ELM。
Mixture-of-Experts for Distributed Edge Computing with Channel-Aware Gating Function
Shenzhen Future Network of Intelligence Institute（深圳未来智能网络研究院）；University of California, San Francisco（加利福尼亚大学旧金山分校）；New Jersey Institute of Technology（新泽西理工学院）；University of Louisiana at Lafayette（路易斯安那大学拉法叶分校）
Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models
School of Computing, National University of Singapore, Singapore.（新加坡国立大学计算机学院）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（细胞系统数学分析卓越研究中心）；School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia.（昆士兰科技大学数学科学学院）；Department of Statistics and Data Science, University of Texas at Austin, Austin, USA（德克萨斯大学奥斯汀分校统计与数据科学系）
N-BEATS-MOE: N-BEATS with a Mixture-of-Experts Layer for Heterogeneous Time Series Forecasting
Faculdade de Engenharia da Universidade do Porto, Porto, Portugal（葡萄牙波尔图大学工程学院）；Laboratory for Artificial Intelligence and Computer Science (LIACC)（人工智能与计算机科学实验室）
Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam（越南胡志明市科学大学数学与计算机科学学院）；Vietnam National University Ho Chi Minh City, Vietnam（越南胡志明市国家大学）；Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（越南胡志明市科学大学信息技术学院）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（细胞系统数学分析 excellence 中心）；School of Mathematical Sciences, Queensland University of Technology, Brisbane City, Australia（昆士兰科技大学数学科学学院）；Department of Statistics and Data Science, University of Texas at Austin, Austin, USA（德克萨斯大学奥斯汀分校统计与数据科学系）
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；INESC-ID（INESC-ID研究所）；Instituto de Telecomunicações（电信研究所）；ELLIS Unit Lisbon（ELLIS里斯本分部）；Gandara AI（Gandara AI公司）
Booth School of Business University of Chicago（芝加哥大学商学院）；Volgenau School of Engineering, George Mason University（乔治·马歇尔大学工程学院）；Department of Systems Engineering and Operations Research George Mason University（乔治·马歇尔大学系统工程与运营管理系）
Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalization
Peter Grünberg Institute, Neuromorphic Software Ecosystems (PGI-15), Jülich Research Centre, Germany（彼得·格林伯格研究所，神经形态软件生态系统（PGI-15），尤利奇研究中心，德国）；Groningen Cognitive Systems and Materials Center (CogniGron), University of Groningen（Groningen认知系统与材料中心（CogniGron）， Groningen大学）；RWTH Aachen University, Aachen, Germany（亚琛RWTH大学，亚琛，德国）；Department of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany（电气工程与信息技术系，亚琛RWTH大学，亚琛，德国）
Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of Queensland（昆士兰大学）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（让·昆廷诺实验室）
Federated Mixture-of-Experts Alignment on Mobile Edge Networks under Data Heterogeneity
Hong Kong JC STEM Lab of Smart City（香港JC STEM实验室）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Department of Electrical and Computer Engineering, The University of Hong Kong（香港大学电子与计算机工程系）
Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment
Fraunhofer Institute for Applied Information Technology FIT（弗劳恩霍夫应用信息技术研究所FIT）；University of Cologne（科隆大学）；University of Stuttgart（斯图加特大学）；University of Southampton（南安普顿大学）；Soochow University（苏州大学）；University Hospital of Cologne（科隆大学医院）
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；ILLC, University of Amsterdam（阿姆斯特丹大学语言、逻辑与计算研究所）；Department of Computational Linguistics, University of Zurich（苏黎世大学计算语言学系）
Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
Department of ECE The Ohio State University（电子工程系，俄亥俄州立大学）；Department of CSE The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Department of Computer Science University of Houston（计算机科学系，休斯顿大学）；Department of ECE Department of CSE The Ohio State University（电子工程系、计算机科学与工程系，俄亥俄州立大学）
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，新加坡科技研究局）；Shanghai University of Engineering Science, China（上海工程技术大学）
GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts
Department of Artificial Intelligence, Lviv Polytechnic National University（人工智能系，利沃夫工业大学）；Department of Electrical Engineering, Eindhoven University of Technology（电气工程系，埃因霍温理工大学）；The Bartlett School of Sustainable Construction, University College London（可持续建设学院，伦敦大学学院）
ELLIS Institute Tübingen（图宾根埃利斯研究所）；LIONS, EPFL（日内瓦理工学院LIONS实验室）；University of Freiburg（弗赖堡大学）；Max-Planck-Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；Prior Labs（Prior实验室）
