MSWA：利用多尺度窗口注意力改进局部注意力
MSWA: Refining Local Attention with Multi-ScaleWindow Attention
AI总结：
本文提出多尺度窗口注意力（MSWA），通过在 Transformer 不同头和层中分配多样化且逐层递增的窗口大小，克服滑动窗口注意力固定窗口的局限，在语言建模和常识推理任务上同时提升效果与效率。
摘要：
基于 Transformer 的大语言模型在广泛的自然语言处理任务中取得了卓越性能。然而，标准自注意力机制存在二次时间复杂度和线性增长的缓存大小问题。滑动窗口注意力（SWA）通过将注意力范围限制在固定大小的局部上下文窗口内来解决这一问题。尽管如此，SWA 在每一层中对每个头都采用统一的窗口大小，导致其在捕获不同尺度上下文时效率低下。为缓解这一局限，我们提出了多尺度窗口注意力（MSWA），它在 Transformer 中跨头和层应用多样化的窗口大小。该方法不仅允许同一层内不同头使用不同的窗口大小，还从浅层到深层逐步增加窗口大小分配，从而使模型能够捕获不同长度和距离的上下文信息。在语言建模和常识推理任务上的实验结果表明，MSWA 在有效性和效率方面均优于传统的局部注意力。
英文摘要：
Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity and linearly increased cache size. Sliding window attention (SWA) solves this problem by restricting the attention range to a fixed-size local context window. Nevertheless, SWA employs a uniform window size for each head in each layer, making it inefficient in capturing context of varying scales. To mitigate this limitation, we propose Multi-Scale Window Attention (MSWA) which applies diverse window sizes across heads and layers in the Transformer. It not only allows for different window sizes among heads within the same layer but also progressively increases window size allocation from shallow to deep layers, thus enabling the model to capture contextual information with different lengths and distances. Experimental results on language modeling and common-sense reasoning tasks substantiate that MSWA outperforms traditional local attention in both effectiveness and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
