Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic compute and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: it combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains from approaches based on local, temporal sparse attention. Our model, the Routing Transformer, endows self-attention with a sparse routing module based on online k-means while reducing the overall complexity of attention to $O\left(n^{1.5}d\right)$ from $O\left(n^2d\right)$ for sequence length $n$ and hidden dimension $d$. We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity) as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers. Additionally, we set a new state-of-the-art on the newly released PG-19 data-set, obtaining a test perplexity of 33.2 with a 22 layer Routing Transformer model trained on sequences of length 8192.
Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
University of California San Diego（加州大学圣迭戈分校）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Fermi National Accelerator Laboratory（费米国家加速器实验室）；University of California Davis（加州大学戴维斯分校）
Deep and wide neural networks successfully fit very complex functions today, but dense models are starting to be prohibitively expensive for inference. To mitigate this, one promising direction is networks that activate a sparse subgraph of the network. The subgraph is chosen by a data-dependent routing function, enforcing a fixed mapping of inputs to subnetworks (e.g., the Mixture of Experts (MoE) paradigm in Switch Transformers). However, prior work is largely empirical, and while existing routing functions work well in practice, they do not lead to theoretical guarantees on approximation ab…
Beijing Institute of Technology（北京理工大学）；Microsoft Corporation（微软公司）；Peking University（北京大学）；Microsoft Research（微软研究院）
Sparse mixture of experts provides larger model capacity while requiring a constant computational overhead. It employs the routing mechanism to distribute input tokens to the best-matched experts according to their hidden representations. However, learning such a routing mechanism encourages token clustering around expert centroids, implying a trend toward representation collapse. In this work, we propose to estimate the routing scores between tokens and experts on a low-dimensional hypersphere. We conduct extensive experiments on cross-lingual language model pre-training and fine-tuning on do…
Recently, Transformer networks have redefined the state of the art in many NLP tasks. However, these models suffer from quadratic computational cost in the input sequence length $n$ to compute pairwise attention in each layer. This has prompted recent research into sparse Transformers that sparsify the connections in the attention layers. While empirically promising for long sequences, fundamental questions remain unanswered: Can sparse Transformers approximate any arbitrary sequence-to-sequence function, similar to their dense counterparts? How does the sparsity pattern and the sparsity level…
When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
Department of Computer Science and Software Engineering（计算机科学与软件工程系）；Auburn University（阿伯拉罕大学）；Department of Information Management（信息管理系）；National Central University（国立中央大学）
Adaptive Routing for Efficient Diffusion Transformer-Based PNI Prediction
Seoul National University（首尔国立大学）；Kyung Hee University（庆熙大学）；OUTTA；Chung-Ang University（Chung-Ang 大学）；Seoul National University School of Medicine（首尔国立大学医学院）；Samsung Changwon Hospital（三星昌原医院）；Samsung Medical Center（三星医疗中心）；NVIDIA AI Technology Center（NVIDIA AI 技术中心）
Recent work on the lottery ticket hypothesis has produced highly sparse Transformers for NMT while maintaining BLEU. However, it is unclear how such pruning techniques affect a model's learned representations. By probing Transformers with more and more low-magnitude weights pruned away, we find that complex semantic information is first to be degraded. Analysis of internal activations reveals that higher layers diverge most over the course of pruning, gradually becoming less complex than their dense counterparts. Meanwhile, early layers of sparse models begin to perform more encoding. Attentio…
EFormer: An Effective Edge-based Transformer for Vehicle Routing Problems
School of Computer Science and Technology, Dalian University of Technology (DUT)（大连理工大学计算机科学与技术学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理学院）；Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology（埃因霍温理工大学工业工程与创新科学系）；Key Laboratory of Social Computing and Cognitive Intelligence (DUT), Ministry of Education, China（社会计算与认知智能重点实验室（DUT））
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fudan University, Shanghai, China（复旦大学）；University of Bath, Bath, United Kingdom（巴斯大学）；Oxford Suzhou Centre for Advanced Research, Suzhou, China（牛津苏滁研究中心）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Department of Computer Science, University of Colorado Boulder, Colorado, USA（计算机科学系，科罗拉多大学博尔德分校）；Department of Electrical Engineering and Computer Science, University of Michigan（电气工程与计算机科学系，密歇根大学）
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Transformers are the mainstream of NLP applications and are becoming increasingly popular in other domains such as Computer Vision. Despite the improvements in model quality, the enormous computation costs make Transformers difficult at deployment, especially when the sequence length is large in emerging applications. Processing attention mechanism as the essential component of Transformer is the bottleneck of execution due to the quadratic complexity. Prior art explores sparse patterns in attention to support long sequence modeling, but those pieces of work are on static or fixed patterns. We…
Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with $L_1$ Prior
Department of Mathematics, University of California, Los Angeles, CA, USA（加州大学洛杉矶分校数学系）；Department of Mathematics, University of South Carolina, Columbia, SC, USA（南卡罗来纳大学数学系）
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；INESC-ID（INESC-ID研究所）；Instituto de Telecomunicações（电信研究所）；ELLIS Unit Lisbon（ELLIS里斯本分部）；Gandara AI（Gandara AI公司）
University of California Irvine（加州大学欧文分校）；Purdue University（普渡大学）；Brookhaven National Laboratory（布鲁克海文国家实验室）；The University of Utah（犹他大学）
We propose GLassoformer, a novel and efficient transformer architecture leveraging group Lasso regularization to reduce the number of queries of the standard self-attention mechanism. Due to the sparsified queries, GLassoformer is more computationally efficient than the standard transformers. On the power grid post-fault voltage prediction task, GLassoformer shows remarkably better prediction than many existing benchmark algorithms in terms of accuracy and stability.
Georgia Institute of Technology（佐治亚理工学院）；Microsoft（微软）
Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational cost. However, SAMs are reported to be parameter inefficient such that larger models do not always lead to better performance. While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i.e., the commonly-used routing methods based on gating mechanisms do not work better than randoml…
