Mixture-of-experts (MoE) models that employ sparse activation have demonstrated effectiveness in significantly increasing the number of parameters while maintaining low computational requirements per token. However, recent studies have established that MoE models are inherently parameter-inefficient as the improvement in performance diminishes with an increasing number of experts. We hypothesize this parameter inefficiency is a result of all experts having equal capacity, which may not adequately meet the varying complexity requirements of different tokens or tasks. In light of this, we propose Stratified Mixture of Experts (SMoE) models, which feature a stratified structure and can assign dynamic capacity to different tokens. We demonstrate the effectiveness of SMoE on three multilingual machine translation benchmarks, containing 4, 15, and 94 language pairs, respectively. We show that SMoE outperforms multiple state-of-the-art MoE models with the same or fewer parameters.
Towards More Effective and Economic Sparsely-Activated Model
Huawei(华为);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Hong Kong University of Science and Technology(香港科技大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Google(谷歌)
用更多的数据、计算和参数来缩放语言模型已经推动了自然语言处理的重大进展。例如,由于可扩展性,GPT-3能够在上下文学习任务上取得很好的结果。然而,训练这些大型密集模型需要大量的计算资源。在本文中,我们提出并开发了一个名为GLaM(Generalist Language Model)的语言模型家族,该语言模型使用稀疏激活的混合专家模型架构来扩展模型容量,同时与密集变体相比,训练成本也大大降低。最大的GLaM有1.2万亿个参数,大约是GPT-3的7倍。它消耗的能量仅为训练GPT-3所用能量的1/3,并且需要一半的计算失败用于推理,同时在29个NLP任务中仍然实现了更好的整体zero-shot和one-shot性能。
CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);University of Science and Technology Beijing(北京科技大学)
MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE(Mohamed bin Zayed人工智能大学(MBZUAI));Michigan State University (MSU), USA(密歇根州立大学(MSU));Amazon Science, UK(亚马逊科学)
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校);The Chinese University of Hong Kong(香港中文大学);MIT(麻省理工学院);Harvard University(哈佛大学)
Continual Pre-training of MoEs: How robust is your router?
Université de Montréal(蒙特利尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);Concordia University(康科迪亚大学);University of Chicago(芝加哥大学);Capital One(Capital One公司)