To be effective, efficient, and diverse, deep learning models need to dynamically choose its architecture based on signals from a population of neurons. We hypothesize dynamic routing models can be improved with neural inhibition in those neural populations. This means signals commonly shared among the various modes of data statistics can be inhibited so that the routing model can choose a specialized expert path for each data sample. Only through inhibition is the routing mechanism able to effectively select neural pathways. We believe this is an under-studied and under-verified implementation methodology for Mixture-of-Experts, dynamic routing, and transformer language models. We provide experimental evidence that the neural inhibition algorithm significantly boosts the performance of general tasks and motivates more effort to be invested in this research direction.
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
AGH University of Krakow, Poland（克拉科夫AGH大学）；Faculty of Mathematics（数学系）；Computer Science, Jagiellonian University, Poland（计算机科学系，杰哥利安大学，波兰）；Centre for Credible Artificial Intelligence, Warsaw University of Technology（可信人工智能中心，华沙技术大学）
MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Experts
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；Institute of Marine Science and Technology, Shandong University（山东大学海洋科学与技术研究院）；Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局下属新加坡制造技术研究院）
Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy (e.g. one resulting in load imbalance) can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choic…
University of Southern California（南加州大学）；AWS AI（亚马逊云科技人工智能部门）
The mixture of Expert (MoE) parallelism is a recent advancement that scales up the model size with constant computational cost. MoE selects different sets of parameters (i.e., experts) for each incoming token, resulting in a sparsely-activated model. Despite several successful applications of MoE, its training efficiency degrades significantly as the number of experts increases. The routing stage in MoE relies on the efficiency of the All2All communication collective, which suffers from network congestion and has poor scalability. To mitigate these issues, we introduce SMILE, which exploits he…
Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization
School of Computing and Artificial Intelligence, Shandong University of Finance and Economics（计算机与人工智能学院，山东财经大学）；School of Computer Science and Technology, Shandong Jianzhu University（计算机科学与技术学院，山东建筑大学）；College of Electronics and Information Engineering, Tongji University（电子信息工程学院，同济大学）；School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学）
The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
The Fin AI（Fin AI）；Georgia Institute of Technology（佐治亚理工学院）；Cornell University（康奈尔大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The University of Manchester（曼彻斯特大学）
The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i.e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference. The routing fluctuation tends to harm sample efficiency because the same input updates different experts but only one is finally used. In this paper, we propose StableMoE with two training stages to address the routing fluctuation…
Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast
Saarland University（萨尔大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）；Bielefeld University（比勒费尔德大学）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
Beijing Institute of Technology（北京理工大学）；Microsoft Corporation（微软公司）；Peking University（北京大学）；Microsoft Research（微软研究院）
Sparse mixture of experts provides larger model capacity while requiring a constant computational overhead. It employs the routing mechanism to distribute input tokens to the best-matched experts according to their hidden representations. However, learning such a routing mechanism encourages token clustering around expert centroids, implying a trend toward representation collapse. In this work, we propose to estimate the routing scores between tokens and experts on a low-dimensional hypersphere. We conduct extensive experiments on cross-lingual language model pre-training and fine-tuning on do…
GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints
School of Computer Science（计算机科学学院）；Georgia Institute of Technology（佐治亚理工学院）；Department of Electrical Engineering（电气工程系）；Tsinghua University（清华大学）；School of Electrical and Computer Engineering（电气与计算机工程学院）
MoxE: Mixture of xLSTM Experts with Entropy-Aware Routing for Efficient Language Modeling
FSM, University of Monastir（FSM，蒙斯塔尔大学）；CES Lab, ENIS, University of Sfax（CES实验室，ENIS，萨菲大学）；Department of Computer Science, College of Computer, Qassim University（计算机科学系，计算机学院，卡西姆大学）；University of Michigan-Flint（密歇根大学弗林特分校）
Towards Generalization-Oriented Models for Vehicle Routing Problems with Mixture-of-Experts
State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology（自主智能无人系统国家重点实验室，北京理工大学）；School of AI, Beijing Institute of Technology（北京理工大学人工智能学院）
