每头计数：无需速度-性能权衡的稀疏注意力
Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off
AI总结：
SPAttention通过引入结构稀疏性范式，优化多头注意力机制，提升计算效率与模型性能。
摘要：
大型语言模型（LLMs）的设计长期以来受到其核心注意力机制内部基本矛盾的阻碍：其卓越的表达能力是建立在计算复杂度为O(H N²)的基础上，该复杂度随着上下文大小（N）平方增长，并与头数（H）线性增长。这种标准实现存在显著的计算冗余，因为所有头都独立地在相同的序列空间上计算注意力。现有的稀疏方法往往以信息完整性换取计算效率。为解决这一效率-性能权衡问题，我们提出了SPAttention，其核心贡献是引入了一种我们称之为
英文摘要：
The design of Large Language Models (LLMs) has long been hampered by a fundamental conflict within their core attention mechanism: its remarkable expressivity is built upon a computational complexity of O(H N^2) that grows quadratically with the context size (N) and linearly with the number of heads (H). This standard implementation harbors significant computational redundancy, as all heads independently compute attention over the same sequence space. Existing sparse methods, meanwhile, often trade information integrity for computational efficiency. To resolve this efficiency-performance trade-off, we propose SPAttention, whose core contribution is the introduction of a new paradigm we term Principled Structural Sparsity. SPAttention does not merely drop connections but instead reorganizes the computational task by partitioning the total attention workload into balanced, non-overlapping distance bands, assigning each head a unique segment. This approach transforms the multi-head attention mechanism from H independent O(N^2) computations into a single, collaborative O(N^2) computation, fundamentally reducing complexity by a factor of H. The structured inductive bias compels functional specialization among heads, enabling a more efficient allocation of computational resources from redundant modeling to distinct dependencies across the entire sequence span. Our work demonstrates that thoughtfully designed structural sparsity can serve as an effective inductive bias that simultaneously improves both computational efficiency and model performance, opening a new avenue for the architectural design of next-generation, high-performance LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
