arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.08906cs.CLcs.LG
收藏

ST-MoE:设计稳定可移植的稀疏专家模型

ST-MoE: Designing Stable and Transferable Sparse Expert Models

Google Brain(谷歌大脑);Google(谷歌);Google Research(谷歌研究院)

Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, William Fedus

摘要:

规模化为自然语言处理开辟了新的领域--但代价高昂。作为响应,已经提出了专家混合(MoE)和切换Transformers作为通向甚至更大和更有能力的语言模型的能量高效路径。但是,在一系列广泛的自然语言任务中推进最先进的技术受到了训练不稳定性和微调过程中质量不确定性的阻碍。我们的工作重点是这些问题,并作为设计指南。我们通过将稀疏模型缩放到269 B参数得出结论,其计算成本与32 B密集编码器-解码器Transformer(稳定和可转移的专家混合或ST-MoE-32 B)相当。稀疏模型首次在迁移学习中实现了最先进的性能,包括推理(SuperGLUE,ARC Easy,ARC Challenge),总结(XSum,CNN-DM),封闭式问答(WebQA,Natural Questions)和对抗性构建任务(Winogrande,ANLI R3)。

英文摘要:

Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models. But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning. Our work focuses on these issues and acts as a design guide. We conclude by scaling a sparse model to 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer (Stable and Transferable Mixture-of-Experts or ST-MoE-32B). For the first time, a sparse model achieves state-of-the-art performance in transfer learning, across a diverse set of tasks including reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum, CNN-DM), closed book question answering (WebQA, Natural Questions), and adversarially constructed tasks (Winogrande, ANLI R3).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑