Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to improve performance, a gated attention model that leverages a gating mechanism within the multi-head self-attention has recently been proposed as a promising alternative. Gated attention has been empirically demonstrated to increase the expressiveness of low-rank mapping in standard attention and even to eliminate the attention sink phenomenon. Despite its efficacy, a clear theoretical understanding of gated attention's benefits remains lacking in the literature. To close this gap, we rigorously show that each entry in a gated attention matrix or a multi-head self-attention matrix can be written as a hierarchical mixture of experts. By recasting learning as an expert estimation problem, we demonstrate that gated attention is more sample-efficient than multi-head self-attention. In particular, while the former needs only a polynomial number of data points to estimate an expert, the latter requires exponentially many data points to achieve the same estimation error. Furthermore, our analysis also provides a theoretical justification for why gated attention yields higher performance when a gate is placed at the output of the scaled dot product attention or the value map rather than at other positions in the multi-head self-attention architecture.
Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of Queensland（昆士兰大学）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（让·昆廷诺实验室）
The sparsely-gated Mixture of Experts (MoE) can magnify a network capacity with a little computational complexity. In this work, we investigate how multi-lingual Automatic Speech Recognition (ASR) networks can be scaled up with a simple routing algorithm in order to achieve better accuracy. More specifically, we apply the sparsely-gated MoE technique to two types of networks: Sequence-to-Sequence Transformer (S2S-T) and Transformer Transducer (T-T). We demonstrate through a set of ASR experiments on multiple language data that the MoE networks can reduce the relative word error rates by 16.3% …
The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational effic…
Universidad Nacional de Colombia（哥伦比亚国立大学）；University of Houston（休斯顿大学）；Instituto Nacional de Astrofísica, Óptica y Electrónica（国家天体物理、光学与电子学研究所）
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality app…
Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models
School of Computing, National University of Singapore, Singapore.（新加坡国立大学计算机学院）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（细胞系统数学分析卓越研究中心）；School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia.（昆士兰科技大学数学科学学院）；Department of Statistics and Data Science, University of Texas at Austin, Austin, USA（德克萨斯大学奥斯汀分校统计与数据科学系）
Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures
University of Science - VNUHCM（越南胡志明市科学大学）；ARC Centre of Excellence for the Mathematical Analysis of Cellular System（细胞系统数学分析卓越研究中心）；School of Mathematical Sciences Queensland University of Technology（昆士兰理工大学数学科学学院）；Department of Statistics University of Michigan（密歇根大学统计学系）；Department of Statistics and Data Science University of Texas at Austin（德克萨斯大学奥斯汀分校统计学与数据科学系）
Dropout is a very effective method in preventing overfitting and has become the go-to regularizer for multi-layer neural networks in recent years. Hierarchical mixture of experts is a hierarchically gated model that defines a soft decision tree where leaves correspond to experts and decision nodes correspond to gating models that softly choose between its children, and as such, the model defines a soft hierarchical partitioning of the input space. In this work, we propose a variant of dropout for hierarchical mixture of experts that is faithful to the tree hierarchy defined by the model, as op…
Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam（越南胡志明市科学大学数学与计算机科学学院）；Vietnam National University Ho Chi Minh City, Vietnam（越南胡志明市国家大学）；Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（越南胡志明市科学大学信息技术学院）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（细胞系统数学分析 excellence 中心）；School of Mathematical Sciences, Queensland University of Technology, Brisbane City, Australia（昆士兰科技大学数学科学学院）；Department of Statistics and Data Science, University of Texas at Austin, Austin, USA（德克萨斯大学奥斯汀分校统计与数据科学系）
Sparsely-gated Mixture of Experts networks (MoEs) have demonstrated excellent scalability in Natural Language Processing. In Computer Vision, however, almost all performant networks are "dense", that is, every input is processed by every parameter. We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks. When applied to image recognition, V-MoE matches the performance of state-of-the-art networks, while requiring as little as half of the compute at inference time. Further, we propose an extension to the routi…
Many data have an underlying dependence on spatial location; it may be weather on the Earth, a simulation on a mesh, or a registered image. Yet this feature is rarely taken advantage of, and violates common assumptions made by many neural network layers, such as translation equivariance. Further, many works that do incorporate locality fail to capture fine-grained structure. To address this, we introduce the Spatial Mixture-of-Experts (SMoE) layer, a sparsely-gated layer that learns spatial structure in the input domain and routes experts at a fine-grained level to utilize it. We also develop …
Gated recurrent units (GRUs) are specialized memory elements for building recurrent neural networks. Despite their incredible success on various tasks, including extracting dynamics underlying neural data, little is understood about the specific dynamics representable in a GRU network. As a result, it is both difficult to know a priori how successful a GRU network will perform on a given task, and also their capacity to mimic the underlying behavior of their biological counterparts. Using a continuous time analysis, we gain intuition on the inner workings of GRU networks. We restrict our prese…
Sparsely Mixture of Experts (MoE) has received great interest due to its promising scaling capability with affordable computational overhead. MoE converts dense layers into sparse experts, and utilizes a gated routing network to make experts conditionally activated. However, as the number of experts grows, MoE with outrageous parameters suffers from overfitting and sparse data allocation. Such problems are especially severe on tasks with limited data, thus hindering the progress for MoE models to improve performance by scaling up. In this work, we propose Mixture of Expert Clusters - a general…
Anomaly detection has been considered under several extents of prior knowledge. Unsupervised methods do not require any labelled data, whereas semi-supervised methods leverage some known anomalies. Inspired by mixture-of-experts models and the analysis of the hidden activations of neural networks, we introduce a novel data-driven anomaly detection method called ARGUE. Our method is not only applicable to unsupervised and semi-supervised environments, but also profits from prior knowledge of self-supervised settings. We designed ARGUE as a combination of dedicated expert networks, which special…
Sparsely gated Mixture-of-Expert (MoE) has demonstrated its effectiveness in scaling up deep neural networks to an extreme scale. Despite that numerous efforts have been made to improve the performance of MoE from the model design or system optimization perspective, existing MoE dispatch patterns are still not able to fully exploit the underlying heterogeneous network environments. In this paper, we propose TA-MoE, a topology-aware routing strategy for large-scale MoE trainging, from a model-system co-design perspective, which can dynamically adjust the MoE dispatch pattern according to the ne…
Mixture of experts (MoE), introduced over 20 years ago, is the simplest gated modular neural network architecture. There is renewed interest in MoE because the conditional computation allows only parts of the network to be used during each inference, as was recently demonstrated in large scale natural language processing models. MoE is also of potential interest for continual learning, as experts may be reused for new tasks, and new experts introduced. The gate in the MoE architecture learns task decompositions and individual experts learn simpler functions appropriate to the gate's decomposit…
Normandie Univ（诺曼底大学）；UNICAEN（卡昂大学）；CNRS（法国国家科学研究中心）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；Grenoble INP（格勒诺布尔国立理工学院）；La Trobe University（乐卓博大学）；University of Queensland（昆士兰大学）
Mixture of experts (MoE) are a popular class of statistical and machine learning models that have gained attention over the years due to their flexibility and efficiency. In this work, we consider Gaussian-gated localized MoE (GLoME) and block-diagonal covariance localized MoE (BLoME) regression models to present nonlinear relationships in heterogeneous data with potential hidden graph-structured interactions between high-dimensional predictors. These models pose difficult statistical estimation and model selection questions, both from a computational and theoretical perspective. This paper is…
University of Queensland（昆士兰大学）；University of Caen（卡昂大学）；La Trobe University（拉筹伯大学）
Mixtures-of-Experts models and their maximum likelihood estimation (MLE) via the EM algorithm have been thoroughly studied in the statistics and machine learning literature. They are subject of a growing investigation in the context of modeling with high-dimensional predictors with regularized MLE. We examine MoE with Gaussian gating network, for clustering and regression, and propose an $\ell_1$-regularized MLE to encourage sparse models and deal with the high-dimensional setting. We develop an EM-Lasso algorithm to perform parameter estimation and utilize a BIC-like criterion to select the m…
Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixtures
Shanghai Center for Mathematical Sciences, Fudan University（复旦大学上海数学中心）；Department of Statistics and Data Science, School of Management, Fudan University（复旦大学管理学院统计与数据科学系）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Washington（华盛顿大学）
Mixture-of-Experts (MoE) is a widely popular model for ensemble learning and is a basic building block of highly successful modern neural networks as well as a component in Gated Recurrent Units (GRU) and Attention networks. However, present algorithms for learning MoE including the EM algorithm, and gradient descent are known to get stuck in local optima. From a theoretical viewpoint, finding an efficient and provably consistent algorithm to learn the parameters remains a long standing open problem for more than two decades. In this paper, we introduce the first algorithm that learns the true…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Washington（华盛顿大学）
Gating is a key feature in modern neural networks including LSTMs, GRUs and sparsely-gated deep neural networks. The backbone of such gated networks is a mixture-of-experts layer, where several experts make regression decisions and gating controls how to weigh the decisions in an input-dependent manner. Despite having such a prominent role in both modern and classical machine learning, very little is understood about parameter recovery of mixture-of-experts since gradient descent and EM algorithms are known to be stuck in local optima in such models. In this paper, we perform a careful analysi…
Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（细胞系统数学分析卓越中心）；School of Mathematical Sciences, Queensland University of Technology（昆士兰科技大学数学科学学院）；Laboratoire d’Analyse et d’Architecture des Systèmes, CNRS（系统分析与架构实验室，CNRS）；School of Medicine and Dentistry, Griffith University（格里菲斯大学医学与牙科学院）；Department of Mathematics and Physical Sciences, La Trobe University（拉特罗布大学数学与物理科学系）；Institute of Mathematics for Industry, Kyushu University（工业数学研究所，九州大学）；Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City（胡志明市大学数学与计算机科学学院）；Vietnam National University, Ho Chi Minh City（越南国家大学，胡志明市）；Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学，Inria，CNRS，格勒诺布尔INP，LJK）
Robust Mixture-of-Expert Training for Convolutional Neural Networks
Michigan State University（密歇根州立大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；MIT（麻省理工学院）；Harvard University（哈佛大学）；UC Santa Barbara（加州大学圣巴巴拉分校）；Carnegie Mellon University（卡内基梅隆大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）
Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts
The University of Queensland（昆士兰大学）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（让·昆汀·莫雷尔实验室）；La Trobe University（拉筹伯大学）；Kyushu University（九州大学）；IRT SystemX
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，新加坡科技研究局）；Shanghai University of Engineering Science, China（上海工程技术大学）
FaST: Efficient and Effective Long-Horizon Forecasting for Large-Scale Spatial-Temporal Graphs via Mixture-of-Experts
Yunnan University（云南大学）；Carnegie Mellon University（卡内基梅隆大学）；Griffith University（格里菲斯大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Beijing Jiaotong University（北京交通大学）
Clinical event sequences in Electronic Health Records (EHRs) record detailed information about the patient condition and patient care as they occur in time. Recent years have witnessed increased interest of machine learning community in developing machine learning models solving different types of problems defined upon information in EHRs. More recently, neural sequential models, such as RNN and LSTM, became popular and widely applied models for representing patient sequence data and for predicting future events or outcomes based on such data. However, a single neural sequential model may not …
