发表机构
RWTH Aachen University; Stanford University（亚琛工业大学; 斯坦福大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究证明在标准复杂性假设下，不存在对所有输入具有非平凡均匀近似保证的次二次注意力近似算法，即使经过多项式预处理亦然，从而确定了均匀注意力近似的计算极限。
AI 中文摘要
Softmax注意力在现代机器学习中无处不在，但其随序列长度呈二次方扩展的特性使其代价高昂。为降低这一成本，注意力常通过快速算法进行近似，这些算法会引入误差，但在实践和某些输入上仍能表现良好。与此同时，注意力应用日益多样化，使得不依赖于特定输入结构的近似保证成为一个有吸引力的目标。对于这种对所有输入的均匀保证，已知的运行时间下界排除了近精确注意力的快速算法，但留下了实践中重要的区间：是否存在一种即使具有适度均匀近似保证的高效算法？我们对此问题给出否定答案。在标准复杂性理论假设下，没有任何真正的次二次算法能够对所有输入均匀地以任何非平凡的加法或相对保证来近似注意力。这种不可能性在最温和的参数区间内成立，而在此区间内，已知算法尚未能在近线性时间内实现强近似保证，并且该结果扩展到实际相关的松弛情形：即使对KV缓存进行多项式预处理，也没有高效算法能够获得非平凡的均匀近似保证，或识别出在稀疏性下获得大量注意力的小规模键集合。总体而言，我们的结果确定了均匀注意力近似的计算极限。
英文摘要
Softmax attention is ubiquitous in modern machine learning, but its quadratic scaling with sequence length makes it costly. To reduce this cost, attention is often approximated with fast algorithms, which incur error but can still perform well in practice and on some inputs. At the same time, the growing diversity of attention applications makes approximation guarantees that do not depend on particular input structure a compelling target. For such uniform guarantees over all inputs, known runtime lower bounds rule out fast algorithms for near-exact attention, but leave open the practically important regime: is there an efficient algorithm with even a modest uniform approximation guarantee? We answer this question negatively. Under standard complexity-theoretic assumptions, no truly subquadratic algorithm can approximate attention with any nontrivial additive or relative guarantee uniformly over all inputs. This impossibility holds in the mildest parameter regime for which known algorithms do not already achieve strong approximation guarantees in near-linear time, and extends to practically relevant relaxations: even after polynomial preprocessing of the KV cache, no efficient algorithm can obtain a nontrivial uniform approximation guarantee, or identify a small set of keys receiving substantial attention under sparsity. Overall, our results settle the computational limits of uniform attention approximation.
