arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2507.04239cs.LGcs.AI
收藏

扩展上下文需要重新思考注意力机制

Scaling Context Requires Rethinking Attention

Manifest AI

Carles Gelada, Jacob Buckman, Sean Zhang, Txus Bach

AI总结:

本文提出幂注意力架构层,以可独立调节状态大小的线性成本序列建模解决Transformer和次二次方架构在长上下文训练中的不足,并通过开源GPU内核实现高效计算,实验证明其在长上下文学习中优于指数和线性注意力。

摘要:

我们认为,无论是Transformer还是次二次方架构都不适合在长序列长度下进行训练:前者的上下文处理成本过高,后者的成本又过低。诸如滑动窗口注意力这类降低Transformer每token成本的方法会损害上下文学习能力,因此同样不适合。为了解决这些局限性,我们提出了幂注意力(power attention),这是一种用于线性成本序列建模的架构层,其状态大小可以独立于参数进行调整,从而在实际领域中释放线性注意力的优势。我们开发并开源了一套用于高效幂注意力的GPU内核,识别出一种新颖的运算融合模式以避免内存和带宽瓶颈。我们在幂注意力的上下文学习实验表明,这些模型在长上下文训练中同时优于指数注意力和线性注意力。

英文摘要:

We argue that neither transformers nor sub-quadratic architectures are well suited to training at long sequence lengths: the cost of processing the context is too expensive in the former, too inexpensive in the latter. Approaches such as sliding window attention which reduce the cost-per-token of a transformer impair in-context learning, and so are also unsuitable. To address these limitations, we introduce power attention, an architectural layer for linear-cost sequence modeling whose state size can be adjusted independently of parameters, unlocking the advantages of linear attention on practical domains. We develop and open-source a set of GPU kernels for efficient power attention, identifying a novel pattern of operation fusion to avoid memory and bandwidth bottlenecks. Our experiments on the in-context learning of power attention shows that these models dominate both exponential attention and linear attention at long-context training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑