SiameseNorm: 突破预规范与后规范之间的障碍
SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm
AI总结：
本文提出SiameseNorm，一种双流架构，通过共享残差块将预规范和后规范结合，从而在保持训练稳定性的同时提升模型性能，适用于多种架构和模态。
摘要：
预规范与后规范之间的长期矛盾仍然是Transformer架构中的一个开放问题，反映了训练稳定性与表示能力之间的根本权衡。先前尝试结合两者优势的研究取得了一定进展，但往往在不同训练设置下表现有限，限制了其更广泛的应用。我们重新审视这一困境，表明单流架构难以协调预规范的稳定身份梯度传播与后规范的主要残差路径归一化。为了解决这种结构张力，我们提出SiameseNorm，一种简单而有效的双流架构，能够与预规范训练配方保持兼容。SiameseNorm通过共享残差块将预规范和后规范流连接起来，允许每个残差块从两个路径接收优化信号，且开销极低。在400M和1.3B密集语言模型、15B MoE模型、视觉Transformer以及扩散Transformer上的大量实验表明，SiameseNorm在各种架构和模态中都能保持强大的训练稳定性的同时提升性能。代码可在https://github.com/Qwen-Applications/SiameseNorm上获得。
英文摘要：
The long-standing tension between Pre- and Post-Norm remains an open problem in Transformer architecture, reflecting a fundamental trade-off between training stability and representational capacity. Prior attempts to combine their strengths have made progress, but often show limited robustness across training settings, restricting their broader applicability. We revisit this dilemma, showing that single-stream architectures struggle to reconcile Pre-Norm's stable identity-gradient propagation with Post-Norm's normalization of the main residual path. To address this structural tension, we propose SiameseNorm, a simple yet effective two-stream architecture that remains compatible with Pre-Norm training recipes. SiameseNorm couples Pre-Norm-like and Post-Norm-like streams through shared residual blocks, allowing each residual block to receive optimization signals from both pathways with negligible overhead. Extensive experiments on 400M and 1.3B dense language models, 15B MoE models, Vision Transformers, and Diffusion Transformers show that SiameseNorm consistently improves performance while maintaining strong training stability across architectures and modalities. Code is available at https://github.com/Qwen-Applications/SiameseNorm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
