Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.
Transformers represent belief state geometry in their residual stream
Simplex（辛普莱克斯）；PIBBSS；Pitzer and Scripps College（皮策与斯克里普斯学院）；University College London（伦敦大学学院）；BITS Principles of Intelligent Behaviour in Biological and Social Systems（BITS生物与社会系统智能行为原理中心）
Transformer是现代NLP模型的支柱。在本文中，我们提出了RealFormer，一个简单而通用的技术来创建剩余注意力层Transformer网络，显着优于规范的Transformer及其变体（BERT，ETC等）。在各种各样的任务上，包括Masked Language Modeling、GLUE、SQuAD、Neural机器翻译、WikiHop、HotpotQA、Natural Questions和OpenKP。我们还根据经验观察到，RealFormer稳定了训练，并导致模型具有更稀疏的注意力。RealFormer的源代码和预先训练的检查点可以在https://github.com/google-research/google-research/tree/master/realformer上找到。
Department of Computer Science, University of California, Irvine, USA（加州大学尔湾分校计算机科学系）；Marshall School of Business, University of Southern California, Los Angeles, USA（南加州大学马歇尔商学院）；Google Research, New York, USA（谷歌研究）
On the Convergence of Gradient Descent on Learning Transformers with Residual Connections
Michigan Institute for Computational Discovery and Engineering, Department of Electrical Engineering and Computer Science, Department of Statistics, University of Michigan（密歇根大学计算发现与工程研究所、电气工程与计算机科学系、统计系）；Department of Computer Science and Engineering, The Ohio State University（俄亥俄州立大学计算机科学与工程系）
Residual Network Based Direct Synthesis of EM Structures: A Study on One-to-One Transformers
School of Electrical and Computer Engineering, Georgia Tech（佐治亚理工学院电气与计算机工程学院）；School of Industrial and Systems Engineering, Georgia Tech（佐治亚理工学院工业与系统工程系）
A Residual Guided strategy with Generative Adversarial Networks in training Physics-Informed Transformer Networks
College of Science, China Agricultural University（中国农业大学理学院）；College of Information and Electrical Engineering, China Agricultural University（中国农业大学信息与电气工程学院）；College of Water Resources and Civil Engineering, China Agricultural University（中国农业大学水利与土木工程学院）
ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；DSC ScaDS.AI, Leipzig University（莱比锡大学DSC ScaDS.AI）；ICT Cluster, Singapore Institute of Technology（新加坡理工大学ICT集群）；Institute for Cancer Genetics and Informatics (ICGI), Oslo, Norway（奥斯陆癌症遗传学与信息学研究所）；Technische Universität Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD – 柏林学习与数据基础研究所）
Forecasting Medium-Horizon Alzheimer's Disease Progression: Residual Gap-Aware Transformers for 24-Month CDR-SB Change from ADNI Clinical and Biomarker Histories
Department of Mathematical Sciences, University of Texas at Dallas（德克萨斯大学达拉斯分校数学科学系）；Department of Plant Science and Landscape Architecture, University of Connecticut（康奈尔大学植物科学与景观建筑系）；Naveen Jindal School of Management, University of Texas at Dallas（德克萨斯大学达拉斯分校纳文·金达管理学院）；Zhejiang Provincial People’s Hospital（浙江省人民医院）
ReCycle: Fast and Efficient Long Time Series Forecasting with Residual Cyclic Transformers
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；German Aerospace Center (DLR)（德国航空航天中心）；INENSUS GmbH（因内森斯有限公司）；Siemens AG, Digital Industries（西门子股份公司数字工业部门）
Residual-based Attention Physics-informed Neural Networks for Spatio-Temporal Ageing Assessment of Transformers Operated in Renewable Power Plants
Mondragon University（蒙dragon大学）；University of the Basque Country (UPV/EHU)（巴斯克大学）；Basque Centre for Applied Mathematics（巴斯克应用数学中心）；KTH, Royal Institute of Technology（皇家理工学院）；Ikerbasque, Basque Foundation for Science（巴斯克科学基金会）
Residual Cross-Attention Transformer-Based Multi-User CSI Feedback with Deep Joint Source-Channel Coding
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；State Key Laboratory of CNS/ATM and the MIIT Key Laboratory of Complex-Field Intelligent Sensing, Beijing Institute of Technology (BIT)（国家 CNS/ATM 工程重点实验室和工信部复杂场智能感知关键实验室，北京理工大学）；Guangzhou Institute of Technology, Xidian University（广州理工大学，西安电子科技大学）
Peking University（北京大学）；Microsoft Research（微软研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；ETH Zurich（苏黎世联邦理工学院）；Tsinghua University（清华大学）
