arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.04477cs.CLcs.AI
收藏

FoundationLayerNorm:将BERT和GPT扩展到1,000层

FoundationLayerNorm: Scaling BERT and GPT to 1,000 Layers

Dezhou Shen

摘要:

主流的BERT/GPT模型只包含10到20层,很少有文献讨论深度BERT/GPT的训练。本文提出了一种简单而有效的方法来稳定BERT和GPT训练。我们成功地将BERT和GPT扩展到1,000层,这比以前的BERT和GPT深了一个数量级。所提出的方法FoundationLayerNormalization能够有效地训练深度神经网络,并在1000层规模下进行了验证。

英文摘要:

The mainstream BERT/GPT model contains only 10 to 20 layers, and there is little literature to discuss the training of deep BERT/GPT. This paper proposes a simple yet effective method to stabilize BERT and GPT training. We successfully scale up BERT and GPT to 1,000 layers, which is an order of magnitude deeper than previous BERT and GPT. The proposed method FoundationLayerNormalization enables efficient training of deep neural networks and is validated at the 1000-layer scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑