TransAxx: Efficient Transformers with Approximate Computing
School of Electrical and Computer Engineering, National Technical University of Athens（电气与计算机工程学院，国家技术大学雅典）；Department of Computer Engineering & Informatics, University of Patras（计算机工程与信息学院，帕特拉大学）；Chair for Embedded Systems at Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院嵌入式系统教授职位）
Vision Transformer (ViT) models which were recently introduced by the transformer architecture have shown to be very competitive and often become a popular alternative to Convolutional Neural Networks (CNNs). However, the high computational requirements of these models limit their practical applicability especially on low-power devices. Current state-of-the-art employs approximate multipliers to address the highly increased compute demands of DNN accelerators but no prior research has explored their use on ViT models. In this work we propose TransAxx, a framework based on the popular PyTorch library that enables fast inherent support for approximate arithmetic to seamlessly evaluate the impact of approximate computing on DNNs such as ViT models. Using TransAxx we analyze the sensitivity of transformer models on the ImageNet dataset to approximate multiplications and perform approximate-aware finetuning to regain accuracy. Furthermore, we propose a methodology to generate approximate accelerators for ViT models. Our approach uses a Monte Carlo Tree Search (MCTS) algorithm to efficiently search the space of possible configurations using a hardware-driven hand-crafted policy. Our evaluation demonstrates the efficacy of our methodology in achieving significant trade-offs between accuracy and power, resulting in substantial gains without compromising on performance.
Recently, Transformer networks have redefined the state of the art in many NLP tasks. However, these models suffer from quadratic computational cost in the input sequence length $n$ to compute pairwise attention in each layer. This has prompted recent research into sparse Transformers that sparsify the connections in the attention layers. While empirically promising for long sequences, fundamental questions remain unanswered: Can sparse Transformers approximate any arbitrary sequence-to-sequence function, similar to their dense counterparts? How does the sparsity pattern and the sparsity level…
Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Tsinghua University（清华大学）；The Simons Institute for the Theory of Computing at the University of California, Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）；University of Pennsylvania（宾夕法尼亚大学）
Learning in Compact Spaces with Approximately Normalized Transformer
University of Freiburg（弗赖堡大学）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Open-Sci Collective（开放科学集体）；LAION；Jülich Supercomputing Centre (JSC)（尤利奇超级计算中心）；Prior Labs（Prior实验室）
University of Wisconsin, Madison（威斯康星大学麦迪逊分校）；American Family Insurance（美国家庭保险公司）
Transformers have emerged as a preferred model for many tasks in natural langugage processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effective ideas include various prespecified sparsity patterns, low-rank basis expansions and combinations thereof. In this paper, we revisit classical Multiresolution Analysis (MRA) concepts such as Wavelets, whose potential value in this setting remains underexplored thus far. We show that simple appro…
Autoregressive Transformers are strong language models but incur O(T) complexity during per-token generation due to the self-attention mechanism. Recent work proposes kernel-based methods to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity. We explore these approaches and find that they are unnecessarily complex, and propose a simple alternative - decaying fast weights - that runs fast on GPU, outperforms prior methods, and retains 99% of attention's performance for GPT-2. We also…
In this work we introduce KERNELIZED TRANSFORMER, a generic, scalable, data driven framework for learning the kernel function in Transformers. Our framework approximates the Transformer kernel as a dot product between spectral feature maps and learns the kernel by learning the spectral distribution. This not only helps in learning a generic kernel end-to-end, but also reduces the time and space complexity of Transformers from quadratic to linear. We show that KERNELIZED TRANSFORMERS achieve performance comparable to existing efficient Transformer architectures, both in terms of accuracy as wel…
Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers
Carnegie Mellon University, Pittsburgh, USA；University of Southern California, Los Angeles, CA, USA；University of California, San Diego, Department of Mathematics, La Jolla, CA, USA；University of California, Davis, Applied Mathematics \& Statistics, Davis, CA, USA；Carnegie Mellon University, Pittsburgh, PA, USA
Transformer has become ubiquitous due to its dominant performance in various NLP and image processing tasks. However, it lacks understanding of how to generate mathematically grounded uncertainty estimates for transformer architectures. Models equipped with such uncertainty estimates can typically improve predictive performance, make networks robust, avoid over-fitting and used as acquisition function in active learning. In this paper, we introduce BayesFormer, a Transformer model with dropouts designed by Bayesian theory. We proposed a new theoretical framework to extend the approximate varia…
Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models. Furthermore, using positional encodings, we circumvent the restriction of permutation equivariance, and show that Transformer models can universally approximate arbitrary continuous sequence-to-sequence functions on a …
Télécom SudParis（南巴黎电信学院）；Institut Polytechnique de Paris（巴黎理工学院）；École Polytechnique（巴黎综合理工学院）；Google Research（谷歌研究院）；DeepMind
This paper introduces the Sequential Monte Carlo Transformer, an original approach that naturally captures the observations distribution in a transformer architecture. The keys, queries, values and attention vectors of the network are considered as the unobserved stochastic states of its hidden structure. This generative model is such that at each time step the received observation is a random function of its past states in a given attention window. In this general state-space setting, we use Sequential Monte Carlo methods to approximate the posterior distributions of the states given the obse…
University of Antwerp（安特卫普大学）；Hasselt University（哈塞尔特大学）
We investigate the power of message-passing neural networks (MPNNs) in their capacity to transform the numerical features stored in the nodes of their input graphs. Our focus is on global expressive power, uniformly over all input graphs, or over graphs of bounded degree with features from a bounded domain. Accordingly, we introduce the notion of a global feature map transformer (GFMT). As a yardstick for expressiveness, we use a basic language for GFMTs, which we call MPLang. Every MPNN can be expressed in MPLang, and our results clarify to which extent the converse inclusion holds. We consid…
The standard normalization method for neural network (NN) models used in Natural Language Processing (NLP) is layer normalization (LN). This is different than batch normalization (BN), which is widely-adopted in Computer Vision. The preferred use of LN in NLP is principally due to the empirical observation that a (naive/vanilla) use of BN leads to significant performance degradation for NLP tasks; however, a thorough understanding of the underlying reasons for this is not always evident. In this paper, we perform a systematic study of NLP transformer models to understand why BN has a poor perf…
Approximation Error Upper and Lower Bounds for Hölder Class with Transformers
School of Mathematics and Statistics, Wuhan University, Wuhan, China（武汉大学数学与统计学学院）；School of Artificial Intelligence, Wuhan University, Wuhan, China（武汉大学人工智能学院）；Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China（湖北省计算科学重点实验室）
How simple can you go? An off-the-shelf transformer approach to molecular dynamics
BIFOLD – Berlin Institute for the Foundations of Learning and Data Machine Learning Group, Technische Universität Berlin（柏林学习与数据基础研究所）；Technische Universität Berlin（柏林技术大学）；Google DeepMind（谷歌DeepMind）；Department of Artificial Intelligence, Korea University（人工智能系，韩国大学）；Max-Planck Institute for Informatics（马克斯·普朗克信息研究所）
Samsung AI Center（三星人工智能中心）；Queen Mary University London（伦敦玛丽女王大学）
This paper tackles the problem of efficient video recognition. In this area, video transformers have recently dominated the efficiency (top-1 accuracy vs FLOPs) spectrum. At the same time, there have been some attempts in the image domain which challenge the necessity of the self-attention operation within the transformer architecture, advocating the use of simpler approaches for token mixing. However, there are no results yet for the case of video recognition, where the self-attention operator has a significantly higher impact (compared to the case of images) on efficiency. To address this ga…
Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly for long sequences, as the standard self-attention mechanism of the Transformer uses $O(n^2)$ time and space with respect to sequence length. In this paper, we demonstrate that the self-attention mechanism can be approximated by a low-rank matrix. We further exploit this finding to propose a new self-attention mechanism, which reduces the overall self-attention complexity from $…
Peking University（北京大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University of Sydney（悉尼大学）；JD Explore Academy（京东探索研究院）
This paper studies the efficiency problem for visual transformers by excavating redundant calculation in given networks. The recent transformer architecture has demonstrated its effectiveness for achieving excellent performance on a series of computer vision tasks. However, similar to that of convolutional neural networks, the huge computational cost of vision transformers is still a severe issue. Considering that the attention mechanism aggregates different patches layer-by-layer, we present a novel patch slimming approach that discards useless patches in a top-down paradigm. We first identif…
Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixtures
Shanghai Center for Mathematical Sciences, Fudan University（复旦大学上海数学中心）；Department of Statistics and Data Science, School of Management, Fudan University（复旦大学管理学院统计与数据科学系）
Efficient Transformers have been developed for long sequence modeling, due to their subquadratic memory and time complexity. Sparse Transformer is a popular approach to improving the efficiency of Transformers by restricting self-attention to locations specified by the predefined sparse patterns. However, leveraging sparsity may sacrifice expressiveness compared to full-attention, when important token correlations are multiple hops away. To combine advantages of both the efficiency of sparse transformer and the expressiveness of full-attention Transformer, we propose \textit{Diffuser}, a new s…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of California, Los Angeles（加州大学洛杉矶分校）
Multi-head attention empowers the recent success of transformers, the state-of-the-art models that have achieved remarkable success in sequence modeling and beyond. These attention mechanisms compute the pairwise dot products between the queries and keys, which results from the use of unnormalized Gaussian kernels with the assumption that the queries follow a mixture of Gaussian distribution. There is no guarantee that this assumption is valid in practice. In response, we first interpret attention in transformers as a nonparametric kernel regression. We then propose the FourierFormer, a new cl…
Music Technology Group, Universitat Pompeu Fabra（庞培法布拉大学音乐技术小组）
We propose a sequence-to-sequence singing synthesizer, which avoids the need for training data with pre-aligned phonetic and acoustic features. Rather than the more common approach of a content-based attention mechanism combined with an autoregressive decoder, we use a different mechanism suitable for feed-forward synthesis. Given that phonetic timings in singing are highly constrained by the musical score, we derive an approximate initial alignment with the help of a simple duration model. Then, using a decoder based on a feed-forward variant of the Transformer model, a series of self-attenti…
Function graph transformers universally approximate operators between function spaces
Doshisha University（大阪市立大学）；RIKEN AIP（日本科学技术厅Advanced Institute for Photonics and Electron器件）；Rice University（里士满大学）；University of Basel（巴塞尔大学）；Simons Chair in Computational and Applied Mathematics and Earth Science（Simons计算与应用数学及地球科学主席职位）；University of Helsinki（赫尔辛基大学）
While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well. Here, we show that Transformer Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.
A common lens to theoretically study neural net architectures is to analyze the functions they can approximate. However, constructions from approximation theory may be unrealistic and therefore less meaningful. For example, a common unrealistic trick is to encode target function values using infinite precision. To address these issues, this work proposes a formal definition of statistically meaningful (SM) approximation which requires the approximating network to exhibit good statistical learnability. We study SM approximation for two function classes: boolean circuits and Turing machines. We …
University of Waterloo（滑铁卢大学）；Vector Institute（向量研究院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
We propose a general deep architecture for learning functions on multiple permutation-invariant sets. We also show how to generalize this architecture to sets of elements of any dimension by dimension equivariance. We demonstrate that our architecture is a universal approximator of these functions, and show superior results to existing methods on a variety of tasks including counting tasks, alignment tasks, distinguishability tasks and statistical distance measurements. This last task is quite important in Machine Learning. Although our approach is quite general, we demonstrate that it can gen…
Epilepsy Seizure Detection and Prediction using an Approximate Spiking Convolutional Transformer
Institute of Neuroinformatics, University of Zurich and ETH Zurich（苏黎世大学与苏黎世联邦理工学院神经信息学研究所）；Delft University of Technology（代尔夫特理工大学）；School of Electronic Science and Technology, Nanjing University（南京大学电子科学与技术学院）
On the Expressive Power of Floating-Point Transformers
Department of Artificial Intelligence, Korea University（韩国大学人工智能系）；Department of Mathematics and Statistics, Sejong University（世宗大学数学与统计学系）；Department of Mathematical Sciences, Gwangju Institute of Science and Technology（全州科学技术研究院数学科学系）
