arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.20637cs.LG
收藏

扩展注意力:一种具有固定数量循环状态的线性时间Transformer的泰勒级数路径

Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States

Zicong Huang

AI总结:

该研究提出一种基于泰勒级数的线性时间Transformer,将注意力指数核展开为n项并压缩为固定循环状态,在长序列下显著降低时间复杂度,最优n=3时精度与成本平衡。

摘要:

我们将注意力的指数核展开为n项的截断泰勒级数,并将每一项分解为张量幂内积,这将因果历史压缩为恰好n个循环状态,每个状态由每个token的一个秩1外积更新,且与上下文长度N无关。该递归相对于截断核是无损的(float64下为2.2×10^-16),且状态在各块间可加(5.7×10^-14),因此预填充(prefill)可采用并行扫描。实测指数为:softmax的时间t∝N^2.28,而状态的时间t∝N^0.91,交叉点在N≈728,当N=8192时加速比为20.8倍;每token解码延迟恒定为7.3μs,而softmax的解码延迟t∝N^1.04,当N=65536时加速比为76.0倍。精度随n呈指数提升,而成本呈组合式增长,因为第m个状态的宽度为组合数C(d+m-1,m)——该边界已通过奇异值分解(SVD)验证,因此最优值为二阶,即n=3,此时误差为6.5×10^-4,每个头有9945个元素。查询和键的归一化是必要前提:若无此操作,自变量会超出收敛半径,增加项数将不再有帮助。

英文摘要:

We expand the exponential kernel of attention into a truncated Taylor series of $n$ terms and factorize each term into a tensor-power inner product, which collapses the causal history into exactly $n$ recurrent states, each updated by one rank-one outer product per token and independent of context length $N$. The recursion is lossless with respect to the truncated kernel ($2.2 \times 10^{-16}$ in float64), and the states are additive across chunks ($5.7 \times 10^{-14}$), so prefill admits a parallel scan. Measured exponents are $t \propto N^{2.28}$ for softmax against $N^{0.91}$ for states, with a crossover at $N \approx 728$ and $20.8\times$ at $N = 8192$; per-token decoding latency is constant at $7.3\,μ$s against $N^{1.04}$ for softmax, giving $76.0\times$ at $N = 65536$. Accuracy improves exponentially in $n$ while cost grows combinatorially, since the $m$-th state is $\binom{d+m-1}{m}$ wide---a bound confirmed by SVD---so the optimum is second order, $n = 3$, at $6.5 \times 10^{-4}$ error and $9{,}945$ elements per head. Query and key normalization is a prerequisite: without it the argument exceeds the convergence radius and adding terms stops helping.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑