用于自然语言处理的顺序注意模块
Sequential Attention Module for Natural Language Processing
摘要：
最近，大型预训练神经语言模型通过微调在许多下游自然语言处理（NLP）应用中获得了显着的性能。在本文中，我们的目标是如何进一步改善语言模型上的令牌表示。因此，我们提出了一个简单而有效的即插即用模块，顺序注意模块（SAM），从预先训练的语言模型学习的令牌嵌入。我们提出的SAM由两个主要的注意力模块依次部署：智能注意力模块（FAM）和令牌式注意力模块（TAM）。更具体地说，FAM可以有效地识别每个维度上特征的重要性，并通过点积对下游NLP应用的原始令牌嵌入进行提升。同时，TAM可以进一步在令牌级别上重新加权特征。此外，我们提出了一个自适应滤波器的FAM，以防止噪声的影响，增加信息的吸收。最后，我们进行了大量的实验，以证明我们提出的SAM的优点和属性。我们首先展示SAM如何在SemEval'21任务7的两个子任务的冠军解决方案中发挥主要作用。之后，我们将SAM应用于情感分析和三个流行的NLP任务，并证明SAM始终优于最先进的基线。
英文摘要：
Recently, large pre-trained neural language models have attained remarkable performance on many downstream natural language processing (NLP) applications via fine-tuning. In this paper, we target at how to further improve the token representations on the language models. We, therefore, propose a simple yet effective plug-and-play module, Sequential Attention Module (SAM), on the token embeddings learned from a pre-trained language model. Our proposed SAM consists of two main attention modules deployed sequentially: Feature-wise Attention Module (FAM) and Token-wise Attention Module (TAM). More specifically, FAM can effectively identify the importance of features at each dimension and promote the effect via dot-product on the original token embeddings for downstream NLP applications. Meanwhile, TAM can further re-weight the features at the token-wise level. Moreover, we propose an adaptive filter on FAM to prevent noise impact and increase information absorption. Finally, we conduct extensive experiments to demonstrate the advantages and properties of our proposed SAM. We first show how SAM plays a primary role in the champion solution of two subtasks of SemEval'21 Task 7. After that, we apply SAM on sentiment analysis and three popular NLP tasks and demonstrate that SAM consistently outperforms the state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
