arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.02790cs.LGcs.CL
收藏

BranchNorm:鲁棒地扩展极深Transformer

BranchNorm: Robustly Scaling Extremely Deep Transformers

Tencent Inc(腾讯公司)

Yijin Liu, Xianfeng Zeng, Fandong Meng, Jie Zhou

AI总结:

针对DeepNorm约束模型更新易导致训练不足的问题,提出BranchNorm方法,按训练阶段动态缩放Transformer非残差分支,兼顾训练稳定性与收敛性能,在多翻译任务上取得更优权衡。

摘要:

近期,DeepNorm将Transformer扩展至极深(即1000层),展现出深度扩展的可观潜力。为稳定深度模型的训练,DeepNorm(Wang等人,2022)试图将模型更新约束为恒定值。尽管施加此类约束对模型训练的早期阶段有益,但可能导致模型在整个训练过程中训练不足。本文提出BranchNorm,它根据训练阶段动态重新缩放Transformer的非残差分支。BranchNorm不仅在理论上能通过早期阶段的平滑梯度范数稳定训练,还能在后续训练阶段促进更好的收敛。在多个翻译任务上的实验结果表明,BranchNorm在训练稳定性与收敛性能之间实现了更优的权衡。

英文摘要:

Recently, DeepNorm scales Transformers into extremely deep (i.e., 1000 layers) and reveals the promising potential of deep scaling. To stabilize the training of deep models, DeepNorm (Wang et al., 2022) attempts to constrain the model update to a constant value. Although applying such a constraint can benefit the early stage of model training, it may lead to undertrained models during the whole training procedure. In this paper, we propose BranchNorm, which dynamically rescales the non-residual branch of Transformer in accordance with the training period. BranchNorm not only theoretically stabilizes the training with smooth gradient norms at the early stage, but also encourages better convergence in the subsequent training stage. Experiment results on multiple translation tasks demonstrate that BranchNorm achieves a better trade-off between training stability and converge performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑