arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1803.02155cs.CL
收藏

相对位置表征的自我注意

Self-Attention with Relative Position Representations

Google(谷歌);Google Brain(谷歌大脑)

Peter Shaw, Jakob Uszkoreit, Ashish Vaswani

摘要:

完全依靠注意力机制,Vaswani等人(2017)引入的Transformer实现了机器翻译的最新成果。与递归和卷积神经网络相比,它没有明确地在其结构中对相对或绝对位置信息进行建模。相反,它需要将绝对位置的表示添加到其输入。在这项工作中,我们提出了一种替代方法,扩展自我注意力机制,有效地考虑表示的相对位置,或序列元素之间的距离。在WMT 2014英语到德语和英语到法语的翻译任务中,这种方法分别比绝对位置表示提高了1.3 BLEU和0.3 BLEU。值得注意的是,我们观察到,结合相对和绝对位置表示不会进一步提高翻译质量。我们描述了一个有效的实现我们的方法,并把它作为一个实例的关系感知自我注意力机制,可以推广到任意图形标记的输入。

英文摘要:

Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑