大型语言模型中动态注意力维护的算法与硬度
Algorithm and Hardness for Dynamic Attention Maintenance in Large Language Models
AI总结:
本文定义了大型语言模型中的动态注意力矩阵乘法问题,提出一种基于惰性更新的数据结构实现条件最优的均摊更新与查询时间,并给出匹配的条件下界证明。
摘要:
大型语言模型(LLMs)从根本上改变了人类生活。注意力机制是所有 LLMs(如 BERT、GPT-1、Transformers、GPT-2、3、3.5 和 4)的关键组件之一。受先前关于注意力乘法问题静态版本的理论研究 [Zandieh, Han, Daliri, and Karbasi arXiv 2023, Alman and Song arXiv 2023] 启发,在本研究中,我们正式定义了注意力矩阵乘法问题的动态版本。存在矩阵 $Q,K, V \in \mathbb{R}^{n \times d}$,它们分别表示 LLMs 中的查询、键和值。在每次迭代中,我们更新 $K$ 或 $V$ 中的一个元素。在查询阶段,我们接收 $(i,j) \in [n] \times [d]$ 作为输入,并希望计算 $(D^{-1} A V)_{i,j}$,其中 $A:=\exp(QK^\top) \in \mathbb{R}^{n \times n}$ 是一个方阵,$D := \mathrm{diag}(A {\bf 1}_n) \in \mathbb{R}^{n \times n}$ 是一个对角矩阵。这里 ${\bf 1}_n$ 表示一个长度为 $n$ 且所有元素均为 1 的向量。\n我们提供了两个结果:一个算法和一个条件下界。\n$\bullet$ 一方面,受 [Demetrescu and Italiano FOCS 2000, Sankowski FOCS 2004, Cohen, Lee and Song STOC 2019, Brand SODA 2020] 中惰性更新思想的启发,我们提供了一种数据结构,其均摊更新时间为 $O(n^{\omega(1,1,\tau)-\tau})$,最坏情况查询时间为 $O(n^{1+\tau})$。\n$\bullet$ 另一方面,我们证明了除非 hinted matrix vector multiplication conjecture [Brand, Nanongkai and Saranurak FOCS 2019] 为假,否则不存在一种算法能同时使用 $O(n^{\omega(1,1,\tau) - \tau- \Omega(1)})$ 的均摊更新时间和 $O(n^{1+\tau-\Omega(1)})$ 的最坏情况查询时间。\n综上所述,除非 hinted matrix vector multiplication conjecture 为假,否则我们的算法结果是条件最优的。
英文摘要:
Large language models (LLMs) have made fundamental changes in human life. The attention scheme is one of the key components over all the LLMs, such as BERT, GPT-1, Transformers, GPT-2, 3, 3.5 and 4. Inspired by previous theoretical study of static version of the attention multiplication problem [Zandieh, Han, Daliri, and Karbasi arXiv 2023, Alman and Song arXiv 2023]. In this work, we formally define a dynamic version of attention matrix multiplication problem. There are matrices $Q,K, V \in \mathbb{R}^{n \times d}$, they represent query, key and value in LLMs. In each iteration we update one entry in $K$ or $V$. In the query stage, we receive $(i,j) \in [n] \times [d]$ as input, and want to answer $(D^{-1} A V)_{i,j}$, where $A:=\exp(QK^\top) \in \mathbb{R}^{n \times n}$ is a square matrix and $D := \mathrm{diag}(A {\bf 1}_n) \in \mathbb{R}^{n \times n}$ is a diagonal matrix. Here ${\bf 1}_n$ denote a length-$n$ vector that all the entries are ones. We provide two results: an algorithm and a conditional lower bound. $\bullet$ On one hand, inspired by the lazy update idea from [Demetrescu and Italiano FOCS 2000, Sankowski FOCS 2004, Cohen, Lee and Song STOC 2019, Brand SODA 2020], we provide a data-structure that uses $O(n^{ω(1,1,τ)-τ})$ amortized update time, and $O(n^{1+τ})$ worst-case query time. $\bullet$ On the other hand, show that unless the hinted matrix vector multiplication conjecture [Brand, Nanongkai and Saranurak FOCS 2019] is false, there is no algorithm that can use both $O(n^{ω(1,1,τ) - τ- Ω(1)})$ amortized update time, and $O(n^{1+τ-Ω(1)})$ worst query time. In conclusion, our algorithmic result is conditionally optimal unless hinted matrix vector multiplication conjecture is false.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。