This paper presents a novel geometric interpretation of LayerNorm and explores how LayerNorm influences the norm and orientation of hidden vectors in the representation space. With these geometric insights, we prepare the foundation for comparing LayerNorm with RMSNorm. We show that the definition of LayerNorm is innately linked to the uniform vector, defined as $\boldsymbol{1} = [1, 1, 1, 1, \cdots, 1]^T \in \mathbb{R}^d$. We then show that the standardization step in LayerNorm can be understood in three simple steps: (i) remove the component of a vector along the uniform vector, (ii) normalize the remaining vector, and (iii) scale the resultant vector by $\sqrt{d}$, where $d$ is the dimensionality of the representation space. We also provide additional insights into how LayerNorm operates at inference time. Finally, we compare the hidden representations of LayerNorm-based LLMs with models trained using RMSNorm and show that all LLMs naturally operate orthogonal to the uniform vector at inference time, that is, on average they do not have a component along the uniform vector during inference. This presents the first mechanistic evidence that removing the component along the uniform vector in LayerNorm is a redundant step. These results advocate for using RMSNorm over LayerNorm which is also more computationally efficient.
MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration
Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece(德摩克利特大学特拉克分校集成电路实验室,电气与计算机工程,德摩克利特大学特拉克分校(DUTH),希腊)
Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE)(复杂与关键软件环境国家重点实验室);Beihang University(北京航空航天大学);Hangzhou International Innovation Institute(杭州国际创新研究院)
Parallel Layer Normalization for Universal Approximation
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE)(复杂与关键软件环境国家重点实验室);Beihang University(北京航空航天大学);Hangzhou International Innovation Institute(杭州国际创新研究院)
Comparative Analysis of CNN and Transformer Architectures with Heart Cycle Normalization for Automated Phonocardiogram Classification
Fraunhofer Institute for Software and Systems Engineering ISST(弗劳恩霍夫软件与系统工程研究所);Department of Computer Science, University of Applied Sciences and Arts Dortmund(应用科学大学多特蒙德计算机科学系);Department of Computer Science, TU Dortmund University(图尔努恩大学计算机科学系);Department of Health Informatics, School of Medicine, Witten/Herdecke University(医学学院健康信息学系,维滕/赫尔德克大学);Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen(医学信息学、生物统计与流行病学研究所(IMIBE),埃森大学医院)