RCMHA：用于自然语言建模的相对卷积多头注意力
RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling
AI总结：
本文提出 RCMHA 框架，通过将深度可分离卷积层应用于 Query、Key、Value 输入嵌入，并在注意力评分阶段引入相对位置编码，在提升自然语言建模准确性的同时降低内存消耗，实验表明其准确率达 0.572，优于 MHA、MDHA 和 RMHA。
摘要：
注意力模块在语言建模中得到普遍应用，并在更广泛的自然语言处理范围内提出了独特的挑战。多头注意力（MHA）采用绝对位置编码，这对 token 长度施加了限制，并且在处理嵌入输入期间消耗大量内存。研究人员目前提出的解决方案涉及使用相对位置编码，类似于 Transformer-XL 或相对多头注意力（RMHA）中所采用的方法，尽管所使用的架构消耗了相当多的内存资源。为了应对这些挑战，本研究致力于改进 MHA，利用相对位置编码与深度可分离卷积层架构相结合，有望在提高准确性的同时最小化内存使用。所提出的 RCMHA 框架需要对两个关键组成部分进行修改：首先，将深度可分离卷积层应用于输入嵌入，包括 Query、Key 和 Value 参数；其次，将相对位置编码纳入注意力评分阶段，并与缩放点积注意力协调集成。实证实验强调了 RCMHA 的优势，其中它表现出更高的准确性，与其他注意力模块（如 MHA、多 DConv 头注意力（MDHA）和 RMHA）相比，获得了 0.572 的分数。在内存利用方面，RMHA 最为节省，平均消耗 2.98 GB，优于需要 3.5 GB 的 RMHA。
英文摘要：
The Attention module finds common usage in language modeling, presenting distinct challenges within the broader scope of Natural Language Processing. Multi-Head Attention (MHA) employs an absolute positional encoding, which imposes limitations on token length and entails substantial memory consumption during the processing of embedded inputs. The current remedy proposed by researchers involves the utilization of relative positional encoding, similar to the approach adopted in Transformer-XL or Relative Multi-Head Attention (RMHA), albeit the employed architecture consumes considerable memory resources. To address these challenges, this study endeavors to refine MHA, leveraging relative positional encoding in conjunction with the Depth-Wise Convolutional Layer architecture, which promises heightened accuracy coupled with minimized memory usage. The proposed RCMHA framework entails the modification of two integral components: firstly, the application of the Depth-Wise Convolutional Layer to the input embedding, encompassing Query, Key, and Value parameters; secondly, the incorporation of Relative Positional Encoding into the attention scoring phase, harmoniously integrated with Scaled Dot-Product Attention. Empirical experiments underscore the advantages of RCMHA, wherein it exhibits superior accuracy, boasting a score of 0.572 in comparison to alternative attention modules such as MHA, Multi-DConv-Head Attention (MDHA), and RMHA. Concerning memory utilization, RMHA emerges as the most frugal, demonstrating an average consumption of 2.98 GB, surpassing RMHA which necessitates 3.5 GB.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
