arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13804cs.AIcs.LGstat.ML
收藏

常数时间注意力:Vashista稀疏注意力用于长上下文解码的指数保证

Attention in Constant Time: Vashista Sparse Attention for Long-Context Decoding with Exponential Guarantees

Datar Consulting(达特咨询)

Vashista Nobaub

AI总结:

Vashista稀疏注意力通过理论保证实现长上下文解码的常数时间效率,平衡精度与计算成本。

摘要:

大型语言模型在推理过程中大部分成本都花在长上下文的注意力上,但经验行为表明,每个查询只有少量的token有意义地贡献。我们通过将注意力建模为凸包上的投影,并分析其熵(softmax-like)松弛来正式化这一现象。我们的主要理论贡献是面稳定性定理,证明在严格互补性边距(由KKT乘数认证的支持间隙(Δ)下,熵注意力集中在常数大小的活跃面:分配给非活跃token的总质量随着(exp(-Ω(Δ/ε)))指数衰减,而活跃面的误差与温度/正则化参数(ε)线性相关。这提供了实际判断稀疏长上下文解码安全性的标准,并提供了一个原则性的调节器,以在准确性和计算量之间进行权衡。基于这些保证,我们引入了Vashista稀疏注意力,这是一种可直接使用的机制,通过页式上下文选择策略维护每个查询的小候选集,与现代推理堆栈兼容。在长上下文评估中,我们观察到稳定的常数大小有效支持、显著的时钟时间加速和在支持间隙诊断预测的范围内质量退化最小。最后,我们讨论了隐私敏感和空气隔离设置下的部署影响,其中可互换的注意力模块可以在不依赖外部检索的情况下实现可预测的延迟和成本。

英文摘要:

Large language models spend most of their inference cost on attention over long contexts, yet empirical behavior suggests that only a small subset of tokens meaningfully contributes to each query. We formalize this phenomenon by modeling attention as a projection onto the convex hull of key vectors and analyzing its entropic (softmax-like) relaxation. Our main theoretical contribution is a face-stability theorem showing that, under a strict complementarity margin (a support gap (Δ) certified by KKT multipliers), entropic attention concentrates on a constant-size active face: the total mass assigned to inactive tokens decays exponentially as (\exp(-Ω(Δ/\varepsilon))), while the error on the active face scales linearly in the temperature/regularization parameter (\varepsilon). This yields a practical criterion for when sparse long-context decoding is safe and provides a principled knob to trade accuracy for compute. Building on these guarantees, we introduce Vashista Sparse Attention, a drop-in mechanism that maintains a small candidate set per query through a paging-style context selection strategy compatible with modern inference stacks. Across long-context evaluations, we observe stable constant-size effective support, strong wall-clock speedups, and minimal quality degradation in the regimes predicted by the support-gap diagnostics. Finally, we discuss deployment implications for privacy-sensitive and air-gapped settings, where interchangeable attention modules enable predictable latency and cost without external retrieval dependencies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑