arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06317cs.LGcs.AIcs.CL
收藏

凝聚定理:Transformer是O(n),而不是O(n²)

Attention-Mass Condensation for Sparse Decoding

NaNZeta LLC(NaNZeta公司)

Jorge L. Ruiz Williams

AI总结:

凝聚定理表明,Transformer的注意力稀疏性是学习到的拓扑属性,而非架构限制,通过动态识别拓扑流形实现无损等价,显著提升推理效率。

摘要:

我们提出了凝聚定理:注意力稀疏性是一种学习到的拓扑属性,而不是架构限制。通过训练语言模型的实证分析,我们发现注意力质量集中在特定的拓扑流形上——并且这个流形可以动态识别而无需检查每个位置。我们证明了一个通用结果:对于任何查询,将注意力投影到凝聚流形(锚点+窗口+动态top-k)可以实现与完整O(n²)注意力100%的输出等价性。这不是近似——这是无损等价。我们验证了这一结论在GPT-2、Pythia、Qwen2、TinyLlama和Mistral上的表现,展示了在1,500多个生成token上的bit精确token匹配。通过将此拓扑映射到硬件,我们的拓扑注意力内核在131,000个token时实现了159倍的测量加速(3.94ms vs 628ms),并在1,000,000个token时预计达到>1,200倍的加速,将推理成本降低超过99.9% compared to Flash Attention。我们得出结论,二次瓶颈是由于幼稚实现造成的,而不是智能。

英文摘要:

Attention-mass concentration creates an opportunity for sparse decoding, but retained mass alone does not guarantee a stable greedy decision: retrieval error, omitted value directions, and recursive decoding all matter. We formalize this distinction with an exact omitted-mass identity and a sufficient downstream margin condition, then characterize a query-dependent mean-pooled block selector. On Qwen2-0.5B, a paired fresh-selection sweep covers supports of 97--769 positions, contexts of 2K--16K, and five prefixes per context. The primary exact-match result is that none of 60 runs remains identical to dense decoding through 128 tokens. Distributional quality is distinct: for supports of at least 193, seven of nine context-support conditions have median teacher-forced continuation perplexity changes within 5\% of dense, but prompt-level ranges include severe 16K outliers. All seven runs with teacher-forced match below 70\% have perplexity increases above 100\%; these observations come from two prefixes and suggest a warning regime, not a general threshold. The measured perplexity is teacher-forced on the dense model's own continuation, not the sparse model's free-running output. Separate retrieval and attention-mass probes illustrate why captured mass alone is not a retrieval or decision guarantee. Isolated operator timings do not establish matched-quality acceleration or end-to-end serving speed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑