arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.12951cs.LGcs.AIcs.CL
收藏

层归一化的几何解释及与 RMSNorm 的比较分析

Geometric Interpretation of Layer Normalization and a Comparative Analysis with RMSNorm

UC Berkeley(加州大学伯克利分校)

Akshat Gupta, Atahan Ozdemir, Gopala Anumanchipalli

AI总结:

本文提出 LayerNorm 的几何解释,证明其标准化步骤包含移除沿均匀向量的分量,并通过实验发现 LLM 推理时隐藏向量自然与均匀向量正交,从而首次提供机制性证据表明该移除步骤冗余,提倡改用计算更高效的 RMSNorm。

摘要:

本文提出了 LayerNorm 的一种新颖几何解释,并探讨了 LayerNorm 如何影响表示空间中隐藏向量的范数与方向。基于这些几何见解,我们为比较 LayerNorm 与 RMSNorm 奠定了基础。我们证明了 LayerNorm 的定义与均匀向量存在内在联系,该均匀向量定义为 $\boldsymbol{1} = [1, 1, 1, 1, \cdots, 1]^T \in \mathbb{R}^d$。随后我们证明,LayerNorm 中的标准化步骤可以理解为三个简单步骤:(i) 移除向量沿均匀向量的分量,(ii) 对剩余向量进行归一化,(iii) 将所得向量缩放 $\sqrt{d}$ 倍,其中 $d$ 为表示空间的维度。我们还对 LayerNorm 在推理时的运作方式提供了额外见解。最后,我们比较了基于 LayerNorm 的 LLM 与使用 RMSNorm 训练的模型的隐藏表示,结果表明所有 LLM 在推理时都自然地在与均匀向量正交的空间中运作,即在推理期间平均而言它们没有沿均匀向量的分量。这提供了首个机制性证据,表明在 LayerNorm 中移除沿均匀向量的分量是冗余步骤。这些结果提倡使用 RMSNorm 替代 LayerNorm,因为前者计算效率更高。

英文摘要:

This paper presents a novel geometric interpretation of LayerNorm and explores how LayerNorm influences the norm and orientation of hidden vectors in the representation space. With these geometric insights, we prepare the foundation for comparing LayerNorm with RMSNorm. We show that the definition of LayerNorm is innately linked to the uniform vector, defined as $\boldsymbol{1} = [1, 1, 1, 1, \cdots, 1]^T \in \mathbb{R}^d$. We then show that the standardization step in LayerNorm can be understood in three simple steps: (i) remove the component of a vector along the uniform vector, (ii) normalize the remaining vector, and (iii) scale the resultant vector by $\sqrt{d}$, where $d$ is the dimensionality of the representation space. We also provide additional insights into how LayerNorm operates at inference time. Finally, we compare the hidden representations of LayerNorm-based LLMs with models trained using RMSNorm and show that all LLMs naturally operate orthogonal to the uniform vector at inference time, that is, on average they do not have a component along the uniform vector during inference. This presents the first mechanistic evidence that removing the component along the uniform vector in LayerNorm is a redundant step. These results advocate for using RMSNorm over LayerNorm which is also more computationally efficient.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑