为何关注一切？关注是关键
Why Attend to Everything? Focus is the Key
AI总结：
本文提出Focus方法，通过学习重要token对来优化注意力机制，实验证明在不同模型规模和架构下均无性能下降，并在速度和质量上优于传统方法。
摘要：
标准注意力机制随序列长度平方增长。高效注意力方法降低了O(n²)成本，但当应用于预训练模型时，通常会降低困惑度、下游准确性或两者。我们引入Focus，一种学习哪些token对重要的方法。Focus添加少量可学习的聚类中心——每层至多148K参数——作为门控：只有属于同一聚类组的token对在长距离上相互关注。Focus是可组合的：只需训练聚类中心即可添加到任何预训练模型中，保持所有原始权重冻结。实验表明，在124M到70B参数的不同模型规模和五种注意力架构上，将Focus组合到预训练模型中不会降低下游基准的性能。令人惊讶的是，稀疏Focus注意力在124M规模上优于完整注意力（30.3 vs. 31.4困惑度），在7B规模上从头训练时与完整注意力相当（13.82 vs. 13.89）。Focus也很快：top-k组成员身份提供了两倍的速度提升，质量优于原始预训练模型。使用我们的FlashAttention分解，Focus在100万token上实现了8.6倍的速度提升，无需自定义内核。
英文摘要：
Standard attention scales quadratically with sequence length. Efficient attention methods reduce this O(n^2) cost, but when retrofitted into pretrained models, they often degrade perplexity, downstream accuracy, or both. We introduce Focus, a method that learns which token pairs matter. Focus adds a small set of learnable centroids--as few as 148K parameters per layer--that act as gates: only token pairs belonging to the same centroid group attend to each other over long ranges. Focus is composable: it can be added to any pretrained model by training only the centroids while keeping all original weights frozen. Experiments show that composing Focus onto pretrained models yields zero degradation on downstream benchmarks across model sizes from 124M to 70B parameters and five attention architectures. Surprisingly, sparse Focus attention outperforms full attention at 124M scale (30.3 vs. 31.4 perplexity) and matches full attention when trained from scratch at 7B scale (13.82 vs. 13.89). Focus is also fast: top-k group membership gives a 2x speedup with better quality than the original pretrained model. Using our FlashAttention decomposition, Focus achieves an 8.6x speedup at 1M tokens without custom kernels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
