NXAI GmbH(NXAI公司);ELLIS Unit(ELLIS单元);Institute for Machine Learning, JKU Linz(约翰内斯·开普勒大学林茨分校机器学习研究所)
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter
In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.
MoxE: Mixture of xLSTM Experts with Entropy-Aware Routing for Efficient Language Modeling
FSM, University of Monastir(FSM,蒙斯塔尔大学);CES Lab, ENIS, University of Sfax(CES实验室,ENIS,萨菲大学);Department of Computer Science, College of Computer, Qassim University(计算机科学系,计算机学院,卡西姆大学);University of Michigan-Flint(密歇根大学弗林特分校)
xLSTM-PINN: Memory-Gated Spectral Remodeling for Physics-Informed Learning
Nanophotonics and Biophotonics Key Laboratory of Jilin Province, School of Physics, Changchun University of Science and Technology(吉林省纳米光子与生物光子重点实验室,物理学院,长春理工大学)
StoxLSTM: A Stochastic Extended Long Short-Term Memory Network for Time Series Forecasting
School of Information and Electronics(信息与电子学院);Beijing Institute of Technology(北京理工大学);Laboratory of Electromagnetic Space Cognition and Intelligent Control(电磁空间认知与智能控制实验室);National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing(空间智能信息处理国家级重点实验室)
A Deep Reinforcement Learning Approach to Automated Stock Trading, using xLSTM Networks
Faculty of Computer Science and Engineering, Shahid Beheshti University(谢赫·贝赫什提大学计算机科学与工程学院);Faculty of Electrical Engineering, Shahid Beheshti University(谢赫·贝赫什提大学电气工程学院)