arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.09862cs.LGcs.AI
收藏

模糊窗口注意力

Blurry Window Attention

Huawei, Zurich, Switzerland(华为技术有限公司(苏黎世));Huawei Advanced Computing and Storage Lab, Shenzhen, China(华为先进计算与存储实验室(深圳))

Axel Laborieux, Christos Sourmpis, Juan Gabriel Kostelec, Qinghai Guo

AI总结:

提出模糊窗口注意力(BLA),一种基于Dirichlet核插值重构模糊KV历史的有界记忆控制方法,在合成任务中状态效率比滑动窗口注意力高8倍,且随状态增大性能提升。

摘要:

Transformer语言模型中的Softmax注意力操作在序列长度上具有二次复杂度,且状态大小以KV缓存形式增长,这成为长上下文场景中的瓶颈。为克服此限制,引入了具有线性复杂度和有限状态大小的替代架构,如状态空间模型(SSM)、线性注意力(LA)和有界记忆控制注意力(ABC)。尽管线性模型在语言困惑度上与Transformer相当,但在需要检索或回忆特定信息的任务中仍落后。本文提出模糊窗口注意力(BLA),一种受SSM启发的新型ABC方法。BLA存储一个频率窗口,通过使用Dirichlet核进行插值从中重建模糊的KV历史。根据Dirichlet核的分辨率,BLA可理解为滑动窗口注意力(SWA)的泛化,或门控槽注意力(GSA)的特例,其中衰减因子由Dirichlet核实现。我们详细描述了BLA的理论和高效实现。在多查询关联回忆(MQAR)合成任务上,我们表明BLA的状态效率比SWA高8倍,且与流行的线性注意力模型竞争;在RegBench合成任务中,在我们测试的线性模型中,只有BLA和SWA随着状态大小增长而提升性能。

英文摘要:

The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a bottleneck in long context scenarios. To overcome this limitation, alternative architectures with linear complexity and finite state size have been introduced, such as State-Space Models (SSMs), Linear Attention (LA), and Attention with Bounded-memory Control (ABC). Though linear models achieve similar language perplexity as Transformers, they are still behind in tasks which require retrieval or recall of specific information. In this work, we introduce Blurry Window Attention (BLA) a novel ABC method inspired by SSMs. BLA stores a frequency window from which a blurry KV history is reconstructed via interpolation using Dirichlet kernels. BLA can be understood as a generalization of Sliding Window Attention (SWA) depending on the Dirichlet kernels resolution or as a special case of the Gated Slot Attention (GSA), where the decay factor is implemented with Dirichlet kernels. We describe in details the theory and efficient implementation of BLA. On the Multi-Query Associate Recall (MQAR) synthetic task, we show that the state efficiency of BLA is 8$\times$ better than SWA and is competitive with popular linear attention models, and in the RegBench synthetic task, only BLA and SWA improve their performance as the state size grows among the linear models we tested.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑