arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.10656cs.LG
收藏

注意力的标记采样复杂度

Token Sample Complexity of Attention

CNRS, ENS Paris, France(法国国家科学研究中心、巴黎高等师范学院)

Léa Bohbot, Cyril Letrouit, Gabriel Peyré, François-Xavier Vialard

AI总结:

研究注意力在极端序列长度下的收敛行为,提出标记采样复杂度概念,分析注意力映射的均匀收敛和变换分布矩的收敛速率,实验验证预测结果。

摘要:

随着大语言模型上下文窗口的扩展,有必要研究注意力在极长序列长度下的行为。我们引入标记采样复杂度:在n个标记上计算的注意力收敛到无限标记极限的速率。我们估计有限n下的收敛界:注意力映射的点wise均匀收敛和变换分布矩的收敛。对于紧支撑(更一般地亚高斯)分布,我们的第一个结果表明,注意力映射在半径为R的球上以速率C(R)/√n收敛,其中C(R)随R指数增长。对于大R,此估计失去实用价值,我们的第二个结果通过建立变换分布矩的收敛速率来解决这一问题。在该情况下,速率是C'(R)/n^β,其中β<1/2,且C'(R)与分布支撑大小的多项式相关。指数β取决于注意力几何和标记分布的谱性质。我们还研究了注意力参数趋于无穷大且softmax趋近于hardmax的 regime,并在此设定下建立了对数收敛速率。合成和真实数据的实验支持我们的预测,并显示预测的减慢在下游准确性中得到反映。

英文摘要:

As context windows in large language models continue to expand, it is essential to characterize how attention behaves at extreme sequence lengths. We introduce token sample complexity: the rate at which attention computed on $n$ tokens converges to its infinite-token limit. We estimate finite-$n$ convergence bounds at two levels: pointwise uniform convergence of the attention map, and convergence of moments for the transformed token distribution. For compactly supported (and more generally sub-Gaussian) distributions, our first result shows that the attention map converges uniformly on a ball of radius $R$ at rate $C(R)/\sqrt{n}$, where $C(R)$ grows exponentially with $R$. For large $R$, this estimate loses practical value, and our second result addresses this issue by establishing convergence rates for the moments of the transformed distribution (the token output of the attention layer). In this case, the rate is $C'(R)/n^β$ with $β<\tfrac{1}{2}$, and $C'(R)$ depends polynomially on the size of the support of the distribution. The exponent $β$ depends on the attention geometry and the spectral properties of the token distribution. We also examine the regime in which the attention parameter tends to infinity and the softmax approaches a hardmax, and in this setting, we establish a logarithmic rate of convergence. Experiments on synthetic and real data support our predictions and show that the predicted slowdown is reflected in downstream accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑