arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.06905cs.CL
收藏

GLaM:基于混合专家模型的语言模型的有效扩展

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Google(谷歌)
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, M… 展开作者

Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, Claire Cui

摘要:

用更多的数据、计算和参数来缩放语言模型已经推动了自然语言处理的重大进展。例如,由于可扩展性,GPT-3能够在上下文学习任务上取得很好的结果。然而,训练这些大型密集模型需要大量的计算资源。在本文中,我们提出并开发了一个名为GLaM(Generalist Language Model)的语言模型家族,该语言模型使用稀疏激活的混合专家模型架构来扩展模型容量,同时与密集变体相比,训练成本也大大降低。最大的GLaM有1.2万亿个参数,大约是GPT-3的7倍。它消耗的能量仅为训练GPT-3所用能量的1/3,并且需要一半的计算失败用于推理,同时在29个NLP任务中仍然实现了更好的整体zero-shot和one-shot性能。

英文摘要:

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑