Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.
Static and multivariate-temporal attentive fusion transformer for readmission risk prediction
School of Computer and Artificial Intelligence, Zhengzhou University(郑州大学计算机与人工智能学院);Cooperative Innovation Center of Internet Healthcare, Zhengzhou University(郑州大学互联网医疗协同创新中心);Fuwai Hospital, National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College(中国医学科学院北京协和医学院国家心血管病中心阜外医院)
MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions
Self-Attentive Ensemble Transformer: Representing Ensemble Interactions in Neural Networks for Earth System Models
Meteorological Institute, University of Hamburg(汉堡大学气象研究所);International Max Planck Research School on Earth System Modelling, Max Planck Institute for Meteorology(马克斯·普朗克气象研究所国际马克斯·普朗克地球系统建模研究学院)
Hierarchical Shift Mixing -- Beyond Dense Attention in Transformers
Dept. of Electrical Engineering, Linköping University, Linköping, Sweden(电子工程系,林霍普大学);RISE Research Institutes of Sweden, Linköping, Sweden(瑞典RISE研究机构)
To Transformers and Beyond: Large Language Models for the Genome
University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(向量人工智能研究所);University Health Network(大学健康网络);Lunenfeld-Tanenbaum Research Institute(卢南菲尔德-塔南鲍姆研究所);University of California, San Francisco(加利福尼亚大学旧金山分校)
Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs
School of Engineering, Westlake University(西湖大学工学院);Department of Computer Science, Yale University(耶鲁大学计算机科学系);Institute of AI Industry Research, Tsinghua University(清华大学人工智能产业研究院)
Self-attentive Transformer for Fast and Accurate Postprocessing of Temperature and Wind Speed Forecasts
Faculty of Applied Engineering, University of Antwerp(应用工程学院,安特卫普大学);CEREA, École des Ponts and EDF R&D(CEREA,Ponts学院和EDF R&D);Royal Meteorological Institute of Belgium(比利时皇家气象研究所);Department of Physics and Astronomy, Ghent University(物理与天文学系,根特大学);United Nations University Institute for Water, Environment and Health(联合国大学水、环境与健康研究所)
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
The Swiss AI Lab, IDSIA(瑞士人工智能实验室,IDSIA);University of Lugano (USI)(卢加诺大学 (USI));SUPSI;King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学 (KAUST))
Attention Beyond Neighborhoods: Reviving Transformer for Graph Clustering
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);Michigan State University(密歇根州立大学);Alibaba Group(阿里巴巴集团)