arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.02582cs.LG
收藏

LayerNorm在Transformer注意力中的表达性作用

On the Expressivity Role of LayerNorm in Transformers' Attention

Carnegie Mellon University(卡内基梅隆大学);Technion(以色列理工学院)

Shaked Brody, Uri Alon, Eran Yahav

AI总结:

本文揭示LayerNorm通过投影和缩放两个几何操作增强Transformer注意力表达性,使其能平等关注所有键且防止键不可选,实验证明其在语言建模和简单函数计算中的有效性。

摘要:

层归一化(LayerNorm)是所有基于Transformer的模型中固有的组成部分。在本文中,我们表明LayerNorm对其后的多头注意力层的表达性至关重要。这与常见的观点相反,即LayerNorm的唯一作用是在前向传播中归一化激活,并在反向传播中归一化梯度。我们考虑LayerNorm的几何解释,并表明它由两个部分组成:(a)将输入向量投影到与[1,1,...,1]向量正交的d-1维空间,以及(b)将所有向量缩放到相同的范数√d。我们表明每个部分对Transformer中其后的注意力层都很重要:(a)投影允许注意力机制创建一个注意力查询,该查询平等地关注所有键,从而将学习此操作的需求从注意力中卸载;以及(b)缩放允许每个键可能获得最高注意力,并防止键变得“不可选择”。我们通过实验表明,Transformer确实在一般语言建模甚至计算简单函数(如“多数”)时受益于LayerNorm的这些属性。我们的代码可在https://github.com/tech-srl/layer_norm_expressivity_role获取。

英文摘要:

Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models. In this paper, we show that LayerNorm is crucial to the expressivity of the multi-head attention layer that follows it. This is in contrast to the common belief that LayerNorm's only role is to normalize the activations during the forward pass, and their gradients during the backward pass. We consider a geometric interpretation of LayerNorm and show that it consists of two components: (a) projection of the input vectors to a $d-1$ space that is orthogonal to the $\left[1,1,...,1\right]$ vector, and (b) scaling of all vectors to the same norm of $\sqrt{d}$. We show that each of these components is important for the attention layer that follows it in Transformers: (a) projection allows the attention mechanism to create an attention query that attends to all keys equally, offloading the need to learn this operation by the attention; and (b) scaling allows each key to potentially receive the highest attention, and prevents keys from being "un-select-able". We show empirically that Transformers do indeed benefit from these properties of LayeNorm in general language modeling and even in computing simple functions such as "majority". Our code is available at https://github.com/tech-srl/layer_norm_expressivity_role .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑