arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.06925cs.CL
收藏

DA-Transformer:距离感知Transformer

DA-Transformer: Distance-aware Transformer

Tsinghua University(清华大学);Microsoft Research Asia(微软亚洲研究院)

Chuhan Wu, Fangzhao Wu, Yongfeng Huang

摘要:

Transformer在NLP领域取得了巨大的成功,它包含了各种先进的模型,如BERT和GPT。然而,Transformer及其现有变体在捕获标记距离方面可能不是最佳的,因为这些方法所使用的位置或距离嵌入通常不能保持真实距离的精确信息,这可能不利于对上下文的顺序和关系进行建模。在本文中,我们提出了DA-Transformer,这是一个距离感知的Transformer,可以利用真实的距离。我们建议将令牌之间的真实距离重新缩放原始的自我注意力权重,这是由注意力查询和键之间的相关性计算。具体地说,在不同的自我注意头的每对令牌之间的相对距离加权不同的可学习参数,控制这些头的长期或短期信息的不同偏好。由于原始的加权实距离可能不是调整自我注意力权重的最佳选择,我们提出了一个可学习的sigmoid函数来将它们映射到具有适当范围的重新缩放系数中。我们首先通过ReLU函数裁剪原始的自我注意力权重,以保持非负性并引入稀疏性,然后将它们与重新缩放的系数相乘,将真实的距离信息编码为自我注意力。在五个基准数据集上的大量实验表明,DA-Transformer可以有效地提高许多任务的性能,并优于vanilla Transformer及其几个变体。

英文摘要:

Transformer has achieved great success in the NLP field by composing various advanced models like BERT and GPT. However, Transformer and its existing variants may not be optimal in capturing token distances because the position or distance embeddings used by these methods usually cannot keep the precise information of real distances, which may not be beneficial for modeling the orders and relations of contexts. In this paper, we propose DA-Transformer, which is a distance-aware Transformer that can exploit the real distance. We propose to incorporate the real distances between tokens to re-scale the raw self-attention weights, which are computed by the relevance between attention query and key. Concretely, in different self-attention heads the relative distance between each pair of tokens is weighted by different learnable parameters, which control the different preferences on long- or short-term information of these heads. Since the raw weighted real distances may not be optimal for adjusting self-attention weights, we propose a learnable sigmoid function to map them into re-scaled coefficients that have proper ranges. We first clip the raw self-attention weights via the ReLU function to keep non-negativity and introduce sparsity, and then multiply them with the re-scaled coefficients to encode real distance information into self-attention. Extensive experiments on five benchmark datasets show that DA-Transformer can effectively improve the performance of many tasks and outperform the vanilla Transformer and its several variants.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑