Mixture of experts (MoE), introduced over 20 years ago, is the simplest gated modular neural network architecture. There is renewed interest in MoE because the conditional computation allows only parts of the network to be used during each inference, as was recently demonstrated in large scale natural language processing models. MoE is also of potential interest for continual learning, as experts may be reused for new tasks, and new experts introduced. The gate in the MoE architecture learns task decompositions and individual experts learn simpler functions appropriate to the gate's decomposition. In this paper: (1) we show that the original MoE architecture and its training method do not guarantee intuitive task decompositions and good expert utilization, indeed they can fail spectacularly even for simple data such as MNIST and FashionMNIST; (2) we introduce a novel gating architecture, similar to attention, that improves performance and results in a lower entropy task decomposition; and (3) we introduce a novel data-driven regularization that improves expert specialization. We empirically validate our methods on MNIST, FashionMNIST and CIFAR-100 datasets.
The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
The Fin AI（Fin AI）；Georgia Institute of Technology（佐治亚理工学院）；Cornell University（康奈尔大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The University of Manchester（曼彻斯特大学）
SD-MoE: Spectral Decomposition for Effective Expert Specialization
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China（复旦大学计算机科学与人工智能学院）；University of Bath, Bath, United Kingdom（巴斯大学）；Oxford Suzhou Centre for Advanced Research, Suzhou, China（牛津苏泽研究中心）；Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Department of Computer Science, University of Colorado Boulder, Colorado, USA（科罗拉多大学博尔德分校计算机科学系）；Research Institute of Tsinghua University in Shenzhen, Shenzhen, China（清华大学深圳研究院）；Greater Bay Area National Center of Technology Innovation, Research Institute of Tsinghua University in Shenzhen, Shenzhen, China（粤港澳大湾区国家技术创新中心，清华大学深圳研究院）；School of Microelectronics, Fudan University, Shanghai, China（复旦大学微电子学院）
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
Department of Computer Science and Engineering, Korea University, Republic of Korea（韩国大学计算机科学与工程系）；KT, Republic of Korea（韩国KT）；School of Software, Soongsil University, Republic of Korea（顺天大学软件学院）
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
Queen Mary University of London（伦敦玛丽女王大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；City University of London（伦敦城市大学）；National and Kapodistrian University of Athens（雅典国立卡波迪斯特里亚大学）；Archimedes AI, Athena RC（阿基米德人工智能公司，雅典娜研究中心）；The Cyprus Institute（塞浦路斯研究所）；Imperial College London（帝国理工学院）
Adaptive Graph Mixture of Residual Experts: Unsupervised Learning on Diverse Graphs with Heterogeneous Specialization
School of New Media and Communication, Tianjin University（新媒体与传播学院，天津大学）；Baidu（百度）；School of Computer Science and Engineering, Beihang University（计算机科学与工程学院，北航大学）
DAMO Academy, Alibaba Group（阿里巴巴集团 DAMO 院）；Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（巴黎-鲁克蒙特研究所）；Rajiv Gandhi University（拉朱·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒研究实验室）
Electric load forecasting is pivotal for power system operation, planning and decision-making. The rise of smart grids and meters has provided more detailed and high-quality load data at multiple levels of granularity, from home to bus and cities. Motivated by similar patterns of loads across different cities in a province in eastern China, in this paper we focus on the Multi-Region Electric Load Forecasting (MRELF) problem, targeting accurate short-term load forecasting for multiple sub-regions within a large region. We identify three challenges for MRELF, including regional variation, contex…
Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learning
Key Laboratory of Cyberspace Big Data Intelligent Security, Ministry of Education（教育部空天地大数据智能安全重点实验室）；School of Artificial Intelligence, Chongqing University of Post and Telecommunications（重庆邮电大学人工智能学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Chongqing Key Laboratory of Brain-Inspired Cognitive Computing and Educational Rehabilitation for Children with Special Needs, Chongqing Normal University（重庆市脑科学与认知计算及特殊儿童教育康复重点实验室）
Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy (e.g. one resulting in load imbalance) can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choic…
MC#: Mixture Compressor for Mixture-of-Experts Large Models
Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；NVIDIA Research（NVIDIA研究）
Harvard University（哈佛大学）；Microsoft Research（微软研究院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Sienna（锡耶纳大学）
We propose Preferential MoE, a novel human-ML mixture-of-experts model that augments human expertise in decision making with a data-based classifier only when necessary for predictive performance. Our model exhibits an interpretable gating function that provides information on when human rules should be followed or avoided. The gating function is maximized for using human-based rules, and classification errors are minimized. We propose solving a coupled multi-objective problem with convex subproblems. We develop approximate algorithms and study their performance and convergence. Finally, we de…
Beihang University（北京航空航天大学）；The Institute of Acoustics of the Chinese Academy of Sciences（中国科学院声学研究所）；Microsoft（微软）
The sparse Mixture-of-Experts (MoE) model is powerful for large-scale pre-training and has achieved promising results due to its model capacity. However, with trillions of parameters, MoE is hard to be deployed on cloud or mobile environment. The inference of MoE requires expert parallelism, which is not hardware-friendly and communication expensive. Especially for resource-limited downstream tasks, such sparse structure has to sacrifice a lot of computing efficiency for limited performance gains. In this work, we observe most experts contribute scarcely little to the MoE fine-tuning and infer…
HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts
Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；INF Technology（INF科技）；Hong Kong University of Science and Technology（香港科技大学）
We develop a mixtures-of-experts (ME) approach to the multiclass classification where the predictors are univariate functions. It consists of a ME model in which both the gating network and the experts network are constructed upon multinomial logistic activation functions with functional inputs. We perform a regularized maximum likelihood estimation in which the coefficient functions enjoy interpretable sparsity constraints on targeted derivatives. We develop an EM-Lasso like algorithm to compute the regularized MLE and evaluate the proposed approach on simulated and real data.
Google, Inc.（谷歌公司）；Courant Institute, NYU（纽约大学库朗研究所）；Facebook AI Group（脸书人工智能团队）
Mixtures of Experts combine the outputs of several "expert" networks, each of which specializes in a different part of the input space. This is achieved by training a "gating" network that maps each input to a distribution over the experts. Such models show promise for building larger networks that are still cheap to compute at test time, and more parallelizable at training time. In this this work, we extend the Mixture of Experts to a stacked model, the Deep Mixture of Experts, with multiple sets of gating and experts. This exponentially increases the number of effective experts by associatin…
Tsinghua University（清华大学）；THiFLY Research（THiFLY研究院）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）
The table-based fact verification task has recently gained widespread attention and yet remains to be a very challenging problem. It inherently requires informative reasoning over natural language together with different numerical and logical reasoning on tables (e.g., count, superlative, comparative). Considering that, we exploit mixture-of-experts and present in this paper a new method: Self-adaptive Mixture-of-Experts Network (SaMoE). Specifically, we have developed a mixture-of-experts neural network to recognize and execute different types of reasoning -- the network is composed of multip…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；The University of Hong Kong（香港大学）
Optimization in multi-task learning (MTL) is more challenging than single-task learning (STL), as the gradient from different tasks can be contradictory. When tasks are related, it can be beneficial to share some parameters among them (cooperation). However, some tasks require additional parameters with expertise in a specific type of data or discrimination (specialization). To address the MTL challenge, we propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad'). This structure allows us to formalize cooperation and specialization as the process of matching expert…
