arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.04517cs.LGcs.AIstat.ML
收藏

xLSTM:扩展型长短期记忆网络

xLSTM: Extended Long Short-Term Memory

NXAI GmbH(NXAI公司);ELLIS Unit(ELLIS单元);Institute for Machine Learning, JKU Linz(约翰内斯·开普勒大学林茨分校机器学习研究所)

Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter

AI总结:

该研究针对传统LSTM在规模化应用上的不足,通过引入指数门控、改进记忆结构提出xLSTM架构,使其在性能和扩展性上可与最先进的Transformer和状态空间模型媲美。

摘要:

20世纪90年代,恒定误差轮播(constant error carousel)与门控机制作为长短期记忆网络(LSTM)的核心思想被提出。此后,LSTM经受住了时间考验,为众多深度学习的成功案例作出了贡献,尤其是它构成了首批大语言模型(LLM)的基础。然而,以可并行自注意力为核心的Transformer技术的出现标志着新时代的开端,在规模化应用上超越了LSTM。我们现在提出一个简单的问题:当将LSTM扩展到数十亿参数规模,利用现代LLM的最新技术,同时缓解LSTM的已知局限性时,我们在语言建模上能达到怎样的水平?首先,我们引入了配备适当归一化与稳定技术的指数门控。其次,我们修改了LSTM的记忆结构,得到两种变体:(i)sLSTM,具备标量记忆、标量更新与新型记忆混合机制;(ii)mLSTM,采用矩阵记忆与协方差更新规则,可完全并行化。将这些LSTM扩展集成到残差块主干中,得到xLSTM块,再通过残差堆叠构建xLSTM架构。指数门控与改进的记忆结构提升了xLSTM的性能,使其在性能和扩展性上都能与最先进的Transformer和状态空间模型相媲美。

英文摘要:

In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑