我们提出了凝聚定理:注意力稀疏性是一种学习到的拓扑属性,而不是架构限制。通过训练语言模型的实证分析,我们发现注意力质量集中在特定的拓扑流形上——并且这个流形可以动态识别而无需检查每个位置。我们证明了一个通用结果:对于任何查询,将注意力投影到凝聚流形(锚点+窗口+动态top-k)可以实现与完整O(n²)注意力100%的输出等价性。这不是近似——这是无损等价。我们验证了这一结论在GPT-2、Pythia、Qwen2、TinyLlama和Mistral上的表现,展示了在1,500多个生成token上的bit精确token匹配。通过将此拓扑映射到硬件,我们的拓扑注意力内核在131,000个token时实现了159倍的测量加速(3.94ms vs 628ms),并在1,000,000个token时预计达到>1,200倍的加速,将推理成本降低超过99.9% compared to Flash Attention。我们得出结论,二次瓶颈是由于幼稚实现造成的,而不是智能。
英文摘要:
Attention-mass concentration creates an opportunity for sparse decoding, but retained mass alone does not guarantee a stable greedy decision: retrieval error, omitted value directions, and recursive decoding all matter. We formalize this distinction with an exact omitted-mass identity and a sufficient downstream margin condition, then characterize a query-dependent mean-pooled block selector. On Qwen2-0.5B, a paired fresh-selection sweep covers supports of 97--769 positions, contexts of 2K--16K, and five prefixes per context. The primary exact-match result is that none of 60 runs remains identical to dense decoding through 128 tokens. Distributional quality is distinct: for supports of at least 193, seven of nine context-support conditions have median teacher-forced continuation perplexity changes within 5\% of dense, but prompt-level ranges include severe 16K outliers. All seven runs with teacher-forced match below 70\% have perplexity increases above 100\%; these observations come from two prefixes and suggest a warning regime, not a general threshold. The measured perplexity is teacher-forced on the dense model's own continuation, not the sparse model's free-running output. Separate retrieval and attention-mass probes illustrate why captured mass alone is not a retrieval or decision guarantee. Isolated operator timings do not establish matched-quality acceleration or end-to-end serving speed.
BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding
Shanghai Institute of Optics and Fine Mechanics(上海光学精密机械研究所);The University of Sydney(悉尼大学);Marquette University(马基特大学);Johns Hopkins University(约翰·霍普金斯大学);Wake Forest University(威克森林大学);University of Texas Health Science Center at Houston(德克萨斯大学健康科学中心休斯顿分部);University of Surrey(萨里大学)
PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding
The Cyprus Institute(塞浦路斯研究所);University of Athens(雅典大学);University of Oxford(牛津大学);Archimedes AI/Athena Research Center(阿基米德AI/雅典娜研究中心);University of Cyprus(塞浦路斯大学)
SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
Microsoft Research(微软研究院);The University of Hong Kong(香港大学);Huazhong University of Science and Technology(华中科技大学);Peking University(北京大学);Tsinghua University(清华大学)
TimeSAE: Causal Sparse Decoding for Faithful Explanations of Black-Box Time Series Models
Telecom Paris(电信巴黎大学);Institut Polytechnique de Paris(巴黎理工学院);Technical University of Munich(慕尼黑技术大学);Helmholtz Munich(海德堡-慕尼黑马克斯·普朗克研究所);Munich Center for Machine Learning(慕尼黑机器学习中心);Munich Data Science Institute(慕尼黑数据科学研究所)
Decoding the Encoding of Functional Brain Networks: an fMRI Classification Comparison of Non-negative Matrix Factorization (NMF), Independent Component Analysis (ICA), and Sparse Coding Algorithms
PySHRED: A Python package for SHallow REcurrent Decoding for sparse sensing, model reduction and scientific discovery
Department of Applied Computational Mathematical Sciences, University of Washington(应用计算数学科学系,华盛顿大学);Department of Computer Science, University of Washington(计算机科学系,华盛顿大学);Department of Mechanical Engineering, University of Washington(机械工程系,华盛顿大学);Department of Energy, Nuclear Engineering Division, Politecnico di Milano(能源部,核工程系,米兰理工学院);Department of Mechanical Engineering, Politecnico di Milano(机械工程系,米兰理工学院);Department of Electrical and Computer Engineering, University of Washington(电气与计算机工程系,华盛顿大学);Department of Applied Mathematics, University of Washington(应用数学系,华盛顿大学)