SpanNorm: 在深度Transformer中协调训练稳定性与性能
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
AI总结：
本文提出SpanNorm技术，通过结合前归一化和后归一化的优势，解决深度Transformer中训练稳定性与性能之间的根本性权衡问题，理论分析和实验结果表明其在密集和专家混合（MoE）场景中均优于传统归一化方案。
摘要：
大型语言模型（LLMs）的成功依赖于深度Transformer架构的稳定训练。一个关键的设计选择是归一化层的位置，导致了一个根本性的权衡：PreNorm架构在深度模型中确保了训练稳定性，但可能牺牲性能；而PostNorm架构提供了强大的性能，但面临严重的训练不稳定性。在本工作中，我们提出SpanNorm，一种新的技术，旨在通过整合两种范式的优点来解决这一困境。结构上，SpanNorm建立了一个跨越整个Transformer块的清晰残差连接以稳定信号传播，同时采用PostNorm风格的计算方式对聚合输出进行归一化以增强模型性能。我们提供了理论分析，证明SpanNorm结合合理的缩放策略可以在整个网络中保持信号方差有界，防止PostNorm模型中出现的梯度问题，并缓解PreNorm中的表示崩溃问题。实验结果表明，SpanNorm在密集和专家混合（MoE）场景中均优于传统归一化方案，为更强大和稳定的Transformer架构铺平了道路。
英文摘要：
The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures. A critical design choice is the placement of normalization layers, leading to a fundamental trade-off: the ``PreNorm'' architecture ensures training stability at the cost of potential performance degradation in deep models, while the ``PostNorm'' architecture offers strong performance but suffers from severe training instability. In this work, we propose SpanNorm, a novel technique designed to resolve this dilemma by integrating the strengths of both paradigms. Structurally, SpanNorm establishes a clean residual connection that spans the entire transformer block to stabilize signal propagation, while employing a PostNorm-style computation that normalizes the aggregated output to enhance model performance. We provide a theoretical analysis demonstrating that SpanNorm, combined with a principled scaling strategy, maintains bounded signal variance throughout the network, preventing the gradient issues that plague PostNorm models, and also alleviating the representation collapse of PreNorm. Empirically, SpanNorm consistently outperforms standard normalization schemes in both dense and Mixture-of-Experts (MoE) scenarios, paving the way for more powerful and stable Transformer architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
