弥合分歧：重新审视Softmax与线性注意力
Bridging the Divide: Reconsidering Softmax and Linear Attention
AI总结：
该研究从单射性和局部建模能力两个核心角度分析线性注意力与Softmax注意力的性能差距，证明补足这两种属性的线性注意力可在多任务上性能超越Softmax注意力且保持更低计算复杂度。
摘要：
Softmax注意力被广泛应用于现代Vision Transformer设计中，能够有效捕获长距离视觉信息，但在处理高分辨率输入时会产生过高的计算成本。相比之下，线性注意力天然具备线性复杂度，在适配更高分辨率图像方面具有巨大潜力。然而，线性注意力的性能表现不尽人意，极大地限制了其在各类场景中的实际应用。本文通过新颖的理论分析进一步缩小线性注意力与Softmax注意力之间的差距，揭示了二者性能差异背后的核心因素。具体而言，我们从两个关键视角来理解并缓解线性注意力的局限性：单射性与局部建模能力。首先，我们证明线性注意力不具备单射性，容易为不同的查询向量分配相同的注意力权重，由于不同查询对应相同输出，会引发严重的语义混淆。其次，我们证实有效的局部建模是Softmax注意力成功的关键，而这正是线性注意力的短板。上述两个根本性差异是导致两种注意力范式存在差距的重要原因，本文中的大量实证验证也证明了这一点。此外，更多实验结果表明，只要赋予线性注意力这两种属性，它就能在各类任务上超越Softmax注意力，同时保持更低的计算复杂度。代码可在https://github.com/LeapLabTHU/InLine获取。
英文摘要：
Widely adopted in modern Vision Transformer designs, Softmax attention can effectively capture long-range visual information; however, it incurs excessive computational cost when dealing with high-resolution inputs. In contrast, linear attention naturally enjoys linear complexity and has great potential to scale up to higher-resolution images. Nonetheless, the unsatisfactory performance of linear attention greatly limits its practical application in various scenarios. In this paper, we take a step forward to close the gap between the linear and Softmax attention with novel theoretical analyses, which demystify the core factors behind the performance deviations. Specifically, we present two key perspectives to understand and alleviate the limitations of linear attention: the injective property and the local modeling ability. Firstly, we prove that linear attention is not injective, which is prone to assign identical attention weights to different query vectors, thus adding to severe semantic confusion since different queries correspond to the same outputs. Secondly, we confirm that effective local modeling is essential for the success of Softmax attention, in which linear attention falls short. The aforementioned two fundamental differences significantly contribute to the disparities between these two attention paradigms, which is demonstrated by our substantial empirical validation in the paper. In addition, more experiment results indicate that linear attention, as long as endowed with these two properties, can outperform Softmax attention across various tasks while maintaining lower computation complexity. Code is available at https://github.com/LeapLabTHU/InLine.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
