arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.14652cs.LGcs.CLstat.ML
收藏

超长Token注意力近似计算的亚线性空间单遍流式算法

One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space

Adobe Research(奥多比研究院);Fuzhou University(福州大学);Sun Yat-sen University(中山大学)

Raghav Addanki, Chenyang Li, Zhao Song, Chiwun Yang

AI总结:

针对超长上下文注意力计算中O(n²)空间瓶颈,提出单遍流式算法,用亚线性空间存储草图矩阵,在内存近乎恒定的同时误差随长度增加而减小,适用于流式LLM部署。

摘要:

注意力计算同时具有$O(n^2)$的时间复杂度和$O(n^2)$的空间复杂度,这使得在涉及长上下文的流式应用中部署大型语言模型(LLMs)需要大量的计算资源。在最近的OpenAI DevDay(2023年11月6日)上,OpenAI发布了一个能够支持128K长度文档的新模型,而在我们的论文中,我们关注的是当上下文长度$n$远大于128K($n \gg 2^d$)时的内存效率问题。考虑一个单层自注意力机制,其中查询、键和值矩阵分别为$Q, K, V \in \mathbb{R}^{n \times d}$,多项式方法近似注意力输出$T \in \mathbb{R}^{n \times d}$。它通过构造$U_1, U_2 \in \mathbb{R}^{n \times t}$来在$n^{1+o(1)}$次时间执行内加速注意力${\sf Attn}(Q, K, V)$的计算。尽管如此,计算近似的注意力矩阵$U_1U_2^\top \in \mathbb{R}^{n \times n}$仍然需要$O(n^2)$的空间,导致显著的内存使用。为了应对这些挑战,我们提出了一种新算法,该算法仅以流式方式读取一遍数据。该方法使用亚线性空间$o(n)$来存储三个草图矩阵,从而避免了对精确$K, V$存储的需求。值得注意的是,我们的算法在处理超长Token时表现出卓越的内存效率。随着Token长度$n$的增加,我们的误差保证会减小,而内存使用几乎保持不变。这一独特属性凸显了我们的技术在流式应用中高效处理LLMs的潜力。

英文摘要:

Attention computation takes both the time complexity of $O(n^2)$ and the space complexity of $O(n^2)$ simultaneously, which makes deploying Large Language Models (LLMs) in streaming applications that involve long contexts requiring substantial computational resources. In recent OpenAI DevDay (Nov 6, 2023), OpenAI released a new model that is able to support a 128K-long document, in our paper, we focus on the memory-efficient issue when context length $n$ is much greater than 128K ($n \gg 2^d$). Considering a single-layer self-attention with Query, Key, and Value matrices $Q, K, V \in \mathbb{R}^{n \times d}$, the polynomial method approximates the attention output $T \in \mathbb{R}^{n \times d}$. It accomplishes this by constructing $U_1, U_2 \in \mathbb{R}^{n \times t}$ to expedite attention ${\sf Attn}(Q, K, V)$ computation within $n^{1+o(1)}$ time executions. Despite this, computing the approximated attention matrix $U_1U_2^\top \in \mathbb{R}^{n \times n}$ still necessitates $O(n^2)$ space, leading to significant memory usage. In response to these challenges, we introduce a new algorithm that only reads one pass of the data in a streaming fashion. This method employs sublinear space $o(n)$ to store three sketch matrices, alleviating the need for exact $K, V$ storage. Notably, our algorithm exhibits exceptional memory-efficient performance with super-long tokens. As the token length $n$ increases, our error guarantee diminishes while the memory usage remains nearly constant. This unique attribute underscores the potential of our technique in efficiently handling LLMs in streaming applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑