DAPE V2：将注意力分数作为特征图进行处理以实现长度外推
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation
AI总结：
本文将Transformer的注意力机制概念化为特征图，应用卷积算子处理相邻注意力分数，解决了由点积表达能力有限导致的长度外推问题，显著提升了模型性能。
摘要：
注意力机制是Transformer模型的基础组件，与早期的前馈神经网络不同，它促进了不同token之间的交互。通常，注意力分数仅由键值-查询乘积决定。然而，本工作将DAPE和NoPE结合的偶然试验表明，在没有位置编码的情况下，在注意力分数上引入额外的MLP，说明经典的键值-查询乘法可能限制了Transformer的性能。在本研究中，我们将注意力概念化为特征图，并应用卷积算子（处理不同头间的相邻注意力分数）来模仿计算机视觉中的处理方法。具体而言，本文的主要贡献是识别并解释了Transformer长度外推问题，将其归因于朴素查询和键值点积的表达能力有限，并成功将长度外推问题转化为一个易于理解的特性图处理问题。这一新颖见解可适应于各种与注意力相关的模型，揭示了当前Transformer架构具有进一步演化的潜力。大量实验表明，将注意力视为特征图并应用卷积作为处理方法，显著提升了Transformer的性能。
英文摘要：
The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens, in contrast to earlier feed-forward neural networks. In general, the attention scores are determined simply by the key-query products. However, this work's occasional trial (combining DAPE and NoPE) of including additional MLPs on attention scores without position encoding indicates that the classical key-query multiplication may limit the performance of Transformers. In this work, we conceptualize attention as a feature map and apply the convolution operator (for neighboring attention scores across different heads) to mimic the processing methods in computer vision. Specifically, the main contribution of this paper is identifying and interpreting the Transformer length extrapolation problem as a result of the limited expressiveness of the naive query and key dot product, and we successfully translate the length extrapolation issue into a well-understood feature map processing problem. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution. Extensive experiments demonstrate that treating attention as a feature map and applying convolution as a processing method significantly enhances Transformer performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
