Transformer is the backbone of modern NLP models. In this paper, we propose RealFormer, a simple and generic technique to create Residual Attention Layer Transformer networks that significantly outperform the canonical Transformer and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. We also observe empirically that RealFormer stabilizes training and leads to models with sparser attention. Source code and pre-trained checkpoints for RealFormer can be found at https://github.com/google-research/google-research/tree/master/realformer.
With the prevailing of mobility as a service (MaaS), it becomes increasingly important to manage multi-traffic modes simultaneously and cooperatively. As an important component of MaaS, short-term passenger flow prediction for multi-traffic modes has thus been brought into focus. It is a challenging problem because the spatiotemporal features of multi-traffic modes are critically complex. Moreover, the passenger flows of multi-traffic modes differentiate and fluctuate significantly. To solve these problems, this paper proposes a multitask learning-based model, called Res-Transformer, for short…
Peking University（北京大学）；Microsoft Research（微软研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；ETH Zurich（苏黎世联邦理工学院）；Tsinghua University（清华大学）
Transformer is a ubiquitous model for natural language processing and has attracted wide attentions in computer vision. The attention maps are indispensable for a transformer model to encode the dependencies among input tokens. However, they are learned independently in each layer and sometimes fail to capture precise patterns. In this paper, we propose a novel and generic mechanism based on evolving attention to improve the performance of transformers. On one hand, the attention maps in different layers share common knowledge, thus the ones in preceding layers can instruct the attention in su…
Residual Cross-Attention Transformer-Based Multi-User CSI Feedback with Deep Joint Source-Channel Coding
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；State Key Laboratory of CNS/ATM and the MIIT Key Laboratory of Complex-Field Intelligent Sensing, Beijing Institute of Technology (BIT)（国家 CNS/ATM 工程重点实验室和工信部复杂场智能感知关键实验室，北京理工大学）；Guangzhou Institute of Technology, Xidian University（广州理工大学，西安电子科技大学）
Transformer layers, which use an alternating pattern of multi-head attention and multi-layer perceptron (MLP) layers, provide an effective tool for a variety of machine learning problems. As the transformer layers use residual connections to avoid the problem of vanishing gradients, they can be viewed as the numerical integration of a differential equation. In this extended abstract, we build upon this connection and propose a modification of the internal architecture of a transformer layer. The proposed model places the multi-head attention sublayer and the MLP sublayer parallel to each other…
Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Beijing Institute of Big Data Research（北京大数据研究院）；Microsoft Research（微软研究院）；Nankai University（南开大学）
The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normaliza…
Department of Computer Science, University of California, Irvine, USA（加州大学尔湾分校计算机科学系）；Marshall School of Business, University of Southern California, Los Angeles, USA（南加州大学马歇尔商学院）；Google Research, New York, USA（谷歌研究）
Transformers represent belief state geometry in their residual stream
Simplex（辛普莱克斯）；PIBBSS；Pitzer and Scripps College（皮策与斯克里普斯学院）；University College London（伦敦大学学院）；BITS Principles of Intelligent Behaviour in Biological and Social Systems（BITS生物与社会系统智能行为原理中心）
On the Convergence of Gradient Descent on Learning Transformers with Residual Connections
Michigan Institute for Computational Discovery and Engineering, Department of Electrical Engineering and Computer Science, Department of Statistics, University of Michigan（密歇根大学计算发现与工程研究所、电气工程与计算机科学系、统计系）；Department of Computer Science and Engineering, The Ohio State University（俄亥俄州立大学计算机科学与工程系）
BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomials
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；Beihang University（北京航空航天大学）
A Residual Guided strategy with Generative Adversarial Networks in training Physics-Informed Transformer Networks
College of Science, China Agricultural University（中国农业大学理学院）；College of Information and Electrical Engineering, China Agricultural University（中国农业大学信息与电气工程学院）；College of Water Resources and Civil Engineering, China Agricultural University（中国农业大学水利与土木工程学院）
School of Electrical and Computer Engineering, Georgia Tech（佐治亚理工学院电气与计算机工程学院）；School of Industrial and Systems Engineering, Georgia Tech（佐治亚理工学院工业与系统工程系）
We propose using machine learning models for the direct synthesis of on-chip electromagnetic (EM) passive structures to enable rapid or even automated designs and optimizations of RF/mm-Wave circuits. As a proof of concept, we demonstrate the direct synthesis of a 1:1 transformer on a 45nm SOI process using our proposed neural network model. Using pre-existing transformer s-parameter files and their geometric design training samples, the model predicts target geometric designs.
ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；DSC ScaDS.AI, Leipzig University（莱比锡大学DSC ScaDS.AI）；ICT Cluster, Singapore Institute of Technology（新加坡理工大学ICT集群）；Institute for Cancer Genetics and Informatics (ICGI), Oslo, Norway（奥斯陆癌症遗传学与信息学研究所）；Technische Universität Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD – 柏林学习与数据基础研究所）
ReCycle: Fast and Efficient Long Time Series Forecasting with Residual Cyclic Transformers
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；German Aerospace Center (DLR)（德国航空航天中心）；INENSUS GmbH（因内森斯有限公司）；Siemens AG, Digital Industries（西门子股份公司数字工业部门）
KingSoft AI Lab（金山人工智能实验室）；Massey University（梅西大学）
Although attention-based Neural Machine Translation has achieved remarkable progress in recent layers, it still suffers from issue of making insufficient use of the output of each layer. In transformer, it only uses the top layer of encoder and decoder in the subsequent process, which makes it impossible to take advantage of the useful information in other layers. To address this issue, we propose a residual tree aggregation of layers for Transformer(RTAL), which helps to fuse information across layers. Specifically, we try to fuse the information across layers by constructing a post-order bin…
