arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.13658cs.CL
收藏

基于相对位置嵌入的Transformer模型改进

Improve Transformer Models with Better Relative Position Embeddings

AWS AI(亚马逊云服务人工智能(AWS AI))

Zhiheng Huang, Davis Liang, Peng Xu, Bing Xiang

摘要:

Transformer体系结构依赖于显式的位置编码,以保持词序的概念。在本文中,我们认为,现有的工作并没有充分利用位置信息。例如,正弦曲线嵌入的初始建议是固定的,不可学习。在本文中,我们首先回顾绝对位置嵌入和相对位置嵌入的现有方法。然后,我们提出了新的技术,鼓励增加查询,关键和相对位置嵌入在自我注意力机制之间的互动。我们最有前途的方法是绝对位置嵌入的推广,与以前的位置嵌入方法相比,SQuAD 1.1的结果有所改善。此外,我们解决的归纳属性的位置嵌入是否可以足够强大,以处理长序列。我们的经验表明,我们的相对位置嵌入方法是合理的推广和强大的归纳的角度来看。最后,我们表明,我们提出的方法可以作为一个近下降的替代,以提高大型模型的精度与一个小的计算预算。

英文摘要:

Transformer architectures rely on explicit position encodings in order to preserve a notion of word order. In this paper, we argue that existing work does not fully utilize position information. For example, the initial proposal of a sinusoid embedding is fixed and not learnable. In this paper, we first review absolute position embeddings and existing methods for relative position embeddings. We then propose new techniques that encourage increased interaction between query, key and relative position embeddings in the self-attention mechanism. Our most promising approach is a generalization of the absolute position embedding, improving results on SQuAD1.1 compared to previous position embeddings approaches. In addition, we address the inductive property of whether a position embedding can be robust enough to handle long sequences. We demonstrate empirically that our relative position embedding method is reasonably generalized and robust from the inductive perspective. Finally, we show that our proposed method can be adopted as a near drop-in replacement for improving the accuracy of large models with a small computational budget.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑