HybridNorm: 一种通过混合归一化实现稳定和高效Transformer训练的方法
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
AI总结：
HybridNorm通过结合Pre-Norm和Post-Norm的优势，提升Transformer训练的稳定性与效率。
摘要：
Transformer架构已成为广泛机器学习任务的首选架构，尤其是在大语言模型（LLMs）中。尽管其表现卓越，但在训练深层Transformer网络时仍存在诸多挑战，尤其是层归一化的位置。虽然Pre-Norm结构由于更强的恒等路径而促进了更稳定的训练，但它们通常导致比Post-Norm更次优的性能。在本文中，我们提出了HybridNorm，一种简单而有效的混合归一化策略，结合了Pre-Norm和Post-Norm的优点。具体而言，HybridNorm在注意力机制中使用QKV归一化，并在每个Transformer块的前馈网络（FFN）中使用Post-Norm。我们提供了理论洞察和实证证据，证明HybridNorm提高了梯度流动和模型鲁棒性。在大规模Transformer模型上进行的广泛实验，包括密集和稀疏变体，显示HybridNorm在多个基准上均优于Pre-Norm和Post-Norm方法。这些发现突显了HybridNorm作为改进深层Transformer模型训练和性能的更稳定和有效技术的潜力。代码可在https://github.com/BryceZhuo/HybridNorm上获取。
英文摘要：
Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, many challenges remain in training deep transformer networks, especially regarding the position of the layer normalization. While Pre-Norm structures facilitate more stable training owing to their stronger identity path, they often lead to suboptimal performance compared to Post-Norm. In this paper, we propose $\textbf{HybridNorm}$, a simple yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm. Specifically, HybridNorm employs QKV normalization within the attention mechanism and Post-Norm in the feed-forward network (FFN) of each transformer block. We provide both theoretical insights and empirical evidence to demonstrate that HybridNorm improves the gradient flow and the model robustness. Extensive experiments on large-scale transformer models, including both dense and sparse variants, show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches across multiple benchmarks. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models. Code is available at https://github.com/BryceZhuo/HybridNorm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
