Transformer models have achieved profound success in prediction tasks in a wide range of applications in natural language processing, speech recognition and computer vision. Extending Transformer's success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer to calibrate its uncertainty. It replaces the scaled dot-product operation with a valid symmetric kernel and uses sparse Gaussian processes (SGP) techniques to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transformers achieve competitive predictive accuracy, while noticeably improving both in-distribution calibration and out-of-distribution robustness and detection.
Texas A&M University（得克萨斯农工大学）；North Carolina State University（北卡罗来纳州立大学）；Purdue University（普渡大学）
Although sparse training has been successfully used in various resource-limited deep learning tasks to save memory, accelerate training, and reduce inference time, the reliability of the produced sparse models remains unexplored. Previous research has shown that deep neural networks tend to be over-confident, and we find that sparse training exacerbates this problem. Therefore, calibrating the sparse models is crucial for reliable prediction and decision-making. In this paper, we propose a new sparse training method to produce sparse models with improved confidence calibration. In contrast to …
Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with $L_1$ Prior
Department of Mathematics, University of California, Los Angeles, CA, USA（加州大学洛杉矶分校数学系）；Department of Mathematics, University of South Carolina, Columbia, SC, USA（南卡罗来纳大学数学系）
Recently, Transformer networks have redefined the state of the art in many NLP tasks. However, these models suffer from quadratic computational cost in the input sequence length $n$ to compute pairwise attention in each layer. This has prompted recent research into sparse Transformers that sparsify the connections in the attention layers. While empirically promising for long sequences, fundamental questions remain unanswered: Can sparse Transformers approximate any arbitrary sequence-to-sequence function, similar to their dense counterparts? How does the sparsity pattern and the sparsity level…
Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper networks, b) the recomputation of attention matrices to save memory, and c) fast attention kernels for training. We call networks with these changes Sparse Transformers, and show they can model sequences tens of thousands of timesteps long using hundreds of layers. We use the same a…
Recent work on the lottery ticket hypothesis has produced highly sparse Transformers for NMT while maintaining BLEU. However, it is unclear how such pruning techniques affect a model's learned representations. By probing Transformers with more and more low-magnitude weights pruned away, we find that complex semantic information is first to be degraded. Analysis of internal activations reveals that higher layers diverge most over the course of pruning, gradually becoming less complex than their dense counterparts. Meanwhile, early layers of sparse models begin to perform more encoding. Attentio…
University of British Columbia（不列颠哥伦比亚大学）；Carnegie Mellon University（卡内基梅隆大学）；Meta；CMU（卡内基梅隆大学）；Google Research（谷歌研究院）；UBC（不列颠哥伦比亚大学）；Amii（阿尔伯塔创新人工智能研究所）
Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called \textbf{Explicit Sparse Transformer}. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing an…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Transformers are the mainstream of NLP applications and are becoming increasingly popular in other domains such as Computer Vision. Despite the improvements in model quality, the enormous computation costs make Transformers difficult at deployment, especially when the sequence length is large in emerging applications. Processing attention mechanism as the essential component of Transformer is the bottleneck of execution due to the quadratic complexity. Prior art explores sparse patterns in attention to support long sequence modeling, but those pieces of work are on static or fixed patterns. We…
University of Warsaw（华沙大学）；Google Research（谷歌研究院）；OpenAI
Large Transformer models yield impressive results on many tasks, but are expensive to train, or even fine-tune, and so slow at decoding that their use and study becomes out of reach. We address this problem by leveraging sparsity. We study sparse variants for all layers in the Transformer and propose Scaling Transformers, a family of next generation Transformer models that use sparse layers to scale efficiently and perform unbatched decoding much faster than the standard Transformer as we scale up the model size. Surprisingly, the sparse layers are enough to obtain the same perplexity as the s…
Many imaging technologies rely on tomographic reconstruction, which requires solving a multidimensional inverse problem given a finite number of projections. Backprojection is a popular class of algorithm for tomographic reconstruction, however it typically results in poor image reconstructions when the projection angles are sparse and/or if the sensors characteristics are not uniform. Several deep learning based algorithms have been developed to solve this inverse problem and reconstruct the image using a limited number of projections. However these algorithms typically require examples of th…
Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic compute and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: it combines the modeling fl…
University of California Irvine（加州大学欧文分校）；Purdue University（普渡大学）；Brookhaven National Laboratory（布鲁克海文国家实验室）；The University of Utah（犹他大学）
We propose GLassoformer, a novel and efficient transformer architecture leveraging group Lasso regularization to reduce the number of queries of the standard self-attention mechanism. Due to the sparsified queries, GLassoformer is more computationally efficient than the standard transformers. On the power grid post-fault voltage prediction task, GLassoformer shows remarkably better prediction than many existing benchmark algorithms in terms of accuracy and stability.
The complex events observed at the NOvA long-baseline neutrino oscillation experiment contain vital information for understanding the most elusive particles in the standard model. The NOvA detectors observe interactions of neutrinos from the NuMI beam at Fermilab. Associating the particles produced in these interaction events to their source particles, a process known as reconstruction, is critical for accurately measuring key parameters of the standard model. Events may contain several particles, each producing sparse high-dimensional spatial observations, and current methods are limited to e…
SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers
University of Pittsburgh（匹兹堡大学）；University of Arizona（亚利桑那大学）；Tongji University（同济大学）；University of Central Florida（中佛罗里达大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
The Transformer has been an indispensable staple in deep learning. However, for real-life applications, it is very challenging to deploy efficient Transformers due to immense parameters and operations of models. To relieve this burden, exploiting sparsity is an effective approach to accelerate Transformers. Newly emerging Ampere GPUs leverage a 2:4 sparsity pattern to achieve model acceleration, while it can hardly meet the diverse algorithm and hardware constraints when deploying models. By contrast, we propose an algorithm-hardware co-optimized framework to flexibly and efficiently accelerat…
TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Qingdao Research Institute, Beihang University（北京航空航天大学青岛研究所）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学（深圳）计算机科学与工程系）；Fert Beijing Research Institute, School of Integrated Circuit Science and Engineering, Beihang University（北京航空航天大学集成电路科学与工程学院）
SSSLab, Dept. of CST China University of Petroleum-Beijing Beijing China（SSSLab，计算机科学与技术系中国石油大学（北京）北京中国）；Baidu Inc. Beijing China（百度公司北京中国）；School of Computer Science Shanghai Jiao Tong University Shanghai China（计算机科学学院上海交通大学上海中国）；China University of Petroleum-Beijing Beihang University Beijing China（中国石油大学（北京）北京航空航天大学北京中国）；China University of Petroleum-Beijing（中国石油大学（北京））；Beihang University（北京航空航天大学）；Baidu Inc.（百度公司）；Shanghai Jiao Tong University（上海交通大学）
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Saratoga High School（萨拉托加高中）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Multimodal classification is a core task in human-centric machine learning. We observe that information is highly complementary across modalities, thus unimodal information can be drastically sparsified prior to multimodal fusion without loss of accuracy. To this end, we present Sparse Fusion Transformers (SFT), a novel multimodal fusion method for transformers that performs comparably to existing state-of-the-art methods while having greatly reduced memory footprint and computation cost. Key to our idea is a sparse-pooling block that reduces unimodal token sets prior to cross-modality modelin…
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Huazhong University of Science and Technology（华中科技大学）；Lehigh University（理海大学）
To capture the semantic graph structure from raw text, most existing summarization approaches are built on GNNs with a pre-trained model. However, these methods suffer from cumbersome procedures and inefficient computations for long-text documents. To mitigate these issues, this paper proposes HETFORMER, a Transformer-based pre-trained model with multi-granularity sparse attentions for long-text extractive summarization. Specifically, we model different types of semantic nodes in raw text as a potential heterogeneous graph and directly learn heterogeneous relationships (edges) among nodes by T…
College of Information Science & Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Baidu Research（百度研究院）；Alibaba Group（阿里巴巴集团）；Department of Computer Science, Binghamton University（宾汉姆顿大学计算机科学系）
In this paper, we focus on the problem of applying the transformer structure to video captioning effectively. The vanilla transformer is proposed for uni-modal language generation task such as machine translation. However, video captioning is a multimodal learning problem, and the video features have much redundancy between different time steps. Based on these concerns, we propose a novel method called sparse boundary-aware transformer (SBAT) to reduce the redundancy in video representation. SBAT employs boundary-aware pooling operation for scores from multihead attention and selects diverse f…
Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis
Rensselaer Polytechnic Institute（伦斯勒理工学院）；New Jersey Institute of Technology（新泽西理工学院）；Michigan State University（密歇根州立大学）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Recently, it has been shown that, in spite of the significant performance of deep neural networks in different fields, those are vulnerable to adversarial examples. In this paper, we propose a gradient-based adversarial attack against transformer-based text classifiers. The adversarial perturbation in our method is imposed to be block-sparse so that the resultant adversarial example differs from the original sentence in only a few words. Due to the discrete nature of textual data, we perform gradient projection to find the minimizer of our proposed optimization problem. Experimental results de…
