层归一化下Transformer的稳定性
Stability of Transformers under Layer Normalization
AI总结：
本文系统性研究不同层归一化放置下Transformer的前向与反向稳定性，推导相关边界、分析梯度反向传播影响，为残差步长缩放提供指导，其框架可用于检查新型Transformer架构的稳定性。
摘要：
尽管Transformer已被广泛应用，但深度Transformer的训练过程可能存在不稳定性。层归一化（Layer Normalization）作为标准组件可提升训练稳定性，但其放置位置常为临时决定。本文针对不同层归一化放置下Transformer的前向传播（隐状态）与反向传播（梯度）稳定性开展系统性研究，所得理论为训练动力学提供关键洞见：训练会使Transformer趋向正则解还是病态行为。针对前向稳定性，本文推导了训练后Transformer中隐状态增长的显式边界；针对反向稳定性，本文分析了层归一化对梯度反向传播的影响，从而解释每种层归一化放置的训练动力学。本文的分析还为Transformer块中残差步长的缩放提供指导，合适的选择可进一步提升稳定性与性能，数值结果证实了理论发现。此外，本文框架为新型架构修改下Transformer的稳定性合理性检查提供了系统性方法，为未来设计提供指导。
英文摘要：
Despite their widespread use, training deep Transformers can be unstable. Layer normalization, a standard component, improves training stability, but its placement has often been ad-hoc. In this paper, we conduct a principled study on the forward (hidden states) and backward (gradient) stability of Transformers under different layer normalization placements. Our theory provides key insights into the training dynamics: whether training drives Transformers toward regular solutions or pathological behaviors. For forward stability, we derive explicit bounds on the growth of hidden states in trained Transformers. For backward stability, we analyze how layer normalization affects the backpropagation of gradients, thereby explaining the training dynamics of each layer normalization placement. Our analysis also guides the scaling of residual steps in Transformer blocks, where appropriate choices can further improve stability and performance. Our numerical results corroborate our theoretical findings. Beyond these results, our framework provides a principled way to sanity-check the stability of Transformers under new architectural modifications, offering guidance for future designs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
