Peri-LN：重新审视Transformer架构中的归一化层
Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
AI总结：
本文提出Peri-LN归一化层放置策略，通过理论分析和高达3.2B参数实验证明其相比Pre-LN和Post-LN具有更均衡的方差增长、更平稳的梯度流动和更稳定的收敛表现。
摘要：
在Transformer中选择一种既能稳定训练又能加速收敛的层归一化（LN）策略仍然十分困难，即使对于当今的大语言模型（LLM）也是如此。我们提出了一个全面的分析基础，用于理解不同LN策略如何影响大规模Transformer中的训练动态。直到最近，Pre-LN和Post-LN长期主导实践，尽管它们在大规模训练中存在局限性。然而，一些开源模型最近开始在没有过多解释的情况下悄然采用第三种策略。这种策略将归一化层放置于子层的外围，我们将这一设计称为Peri-LN。虽然Peri-LN已经展现出有前景的性能，但其精确机制和优势几乎仍未得到探索。我们的深入分析刻画了不同LN策略的独特行为，展示了每种放置方式如何塑造激活方差和梯度传播。为了验证我们的理论见解，我们在参数量高达$3.2$B的Transformer上进行了广泛实验，结果表明Peri-LN始终能够实现更均衡的方差增长、更平稳的梯度流动以及收敛稳定性。我们的结果表明，Peri-LN值得在大规模Transformer架构中得到更广泛的考虑，为LN的最优放置提供了新的见解。
英文摘要：
Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
