arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.02377cs.CLcs.LG
收藏

PermuteFormer:有效的长序列相对位置编码

PermuteFormer: Efficient Relative Position Encoding for Long Sequences

Peking University(北京大学)

Peng Chen

摘要:

Transformer,Performer的最新变体使用线性注意力机制将Transformer扩展到更长的序列。然而,它与相对位置编码不兼容,相对位置编码优于绝对位置编码。在本文中,我们讨论了可能的方式来添加相对位置编码的表演者。基于分析,我们提出了PermuteFormer,一个基于性能的模型,具有相对位置编码,在长序列上线性缩放。PermuteFormer对查询和键应用位置相关的转换,将位置信息编码到attention模块中。这种转换是精心制作的,因此自我注意的最终输出不受标记的绝对位置的影响。PermuteFormer通过设计引入了可以忽略的计算开销,其运行速度与Performer一样快。我们评估PermuteFormer的长距离竞技场,长序列的数据集,以及WikiText-103,语言建模数据集。实验结果表明,PermuteFormer在几乎没有计算开销的情况下均匀地提高了Performer的性能,并且在大多数任务上的性能优于vanilla Transformer。

英文摘要:

A recent variation of Transformer, Performer, scales Transformer to longer sequences with a linear attention mechanism. However, it is not compatible with relative position encoding, which has advantages over absolute position encoding. In this paper, we discuss possible ways to add relative position encoding to Performer. Based on the analysis, we propose PermuteFormer, a Performer-based model with relative position encoding that scales linearly on long sequences. PermuteFormer applies position-dependent transformation on queries and keys to encode positional information into the attention module. This transformation is carefully crafted so that the final output of self-attention is not affected by absolute positions of tokens. PermuteFormer introduces negligible computational overhead by design that it runs as fast as Performer. We evaluate PermuteFormer on Long-Range Arena, a dataset for long sequences, as well as WikiText-103, a language modeling dataset. The experiments show that PermuteFormer uniformly improves the performance of Performer with almost no computational overhead and outperforms vanilla Transformer on most of the tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑