arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.06104cs.CL
收藏

Transformers是多状态RNNs

Transformers are Multi-State RNNs

The Hebrew University of Jerusalem(耶路撒冷希伯来大学);FAIR, AI at Meta(Meta AI FAIR部门)

Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, Roy Schwartz

AI总结:

本研究证明仅解码器transformers可视为无界多状态RNNs,提出无需训练的TOVA压缩策略压缩键值缓存,实验显示其性能接近完整模型,缓存仅需原大小1/8时吞吐量提升4.8倍,兼具理论与实用价值。

摘要:

Transformers在概念上被认为与上一代最先进的NLP模型——循环神经网络(RNNs)不同。在本研究中,我们证明仅解码器的transformers实际上可以被概念化为无界多状态RNNs——一种隐藏状态大小不受限的RNN变体。我们进一步表明,通过固定transformers的隐藏状态大小,可将其转换为有界多状态RNNs,从而有效压缩其键值缓存。我们提出了一种新颖的、无需训练的压缩策略——基于注意力的令牌省略(Token Omission Via Attention,TOVA)。我们在四项长距离任务和多个大语言模型(LLMs)上开展的实验表明,TOVA的性能优于多种基线压缩策略。值得注意的是,我们的结果几乎与完整模型相当,在某些情况下仅使用原始缓存大小的1/8,这意味着吞吐量提升了4.8倍。我们的研究结果阐明了transformers与RNNs之间的联系,并有助于缓解LLMs最棘手的计算瓶颈之一——键值缓存的大小问题。我们在https://github.com/schwartz-lab-NLP/TOVA公开发布了代码。

英文摘要:

Transformers are considered conceptually different from the previous generation of state-of-the-art NLP models - recurrent neural networks (RNNs). In this work, we demonstrate that decoder-only transformers can in fact be conceptualized as unbounded multi-state RNNs - an RNN variant with unlimited hidden state size. We further show that transformers can be converted into $\textit{bounded}$ multi-state RNNs by fixing the size of their hidden state, effectively compressing their key-value cache. We introduce a novel, training-free compression policy - $\textbf{T}$oken $\textbf{O}$mission $\textbf{V}$ia $\textbf{A}$ttention (TOVA). Our experiments with four long range tasks and several LLMs show that TOVA outperforms several baseline compression policies. Particularly, our results are nearly on par with the full model, using in some cases only $\frac{1}{8}$ of the original cache size, which translates to 4.8X higher throughput. Our results shed light on the connection between transformers and RNNs, and help mitigate one of LLMs' most painful computational bottlenecks - the size of their key-value cache. We publicly release our code at https://github.com/schwartz-lab-NLP/TOVA

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑