The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers - particularly with respect to sequence length - have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively. In this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers.
Fixed-Point RNNs: Interpolating from Diagonal to Dense
ELLIS Institute Tuebingen（图宾根ELLIS研究所）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；Department of Mathematics, Imperial College London（伦敦帝国学院数学系）
pLSTM: parallelizable Linear Source Transition Mark networks
ELLIS Unit Linz（林茨ELLIS单位）；LIT AI Lab（LIT人工智能实验室）；Institute for Machine Learning（机器学习研究所）；Johannes Kepler University Linz（约翰·凯撒大学林茨分校）；NXAI GmbH（NXAI公司）
Adaptive AI Research Group, University of Lübeck, Germany（适应性人工智能研究组，吕贝克大学）；Neuro-Cognitive Modeling Group, University of Tübingen, Germany（神经认知建模组，图宾根大学）
Focused Hierarchical RNNs for Conditional Sequence Processing
Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）；Polytechnique Montreal（蒙特利尔高等理工学院）；Microsoft Research（微软研究院）；Jagiellonian University（雅盖隆大学）；University of Montreal（蒙特利尔大学）；Facebook AI Research（Facebook人工智能研究院）；McGill University（麦吉尔大学）；HEC Montreal（蒙特利尔高等商学院）
