Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models. But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning. Our work focuses on these issues and acts as a design guide. We conclude by scaling a sparse model to 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer (Stable and Transferable Mixture-of-Experts or ST-MoE-32B). For the first time, a sparse model achieves state-of-the-art performance in transfer learning, across a diverse set of tasks including reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum, CNN-DM), closed book question answering (WebQA, Natural Questions), and adversarially constructed tasks (Winogrande, ANLI R3).
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
University of Warsaw(华沙大学);Wroclaw University of Science and Technology(弗罗茨瓦夫理工大学);Institute of Mathematics, Polish Academy of Sciences(波兰科学院数学研究所);Institute of Fundamental Technological Research, Polish Academy of Sciences(波兰科学院基础技术研究所)
Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);Ant Group(蚂蚁集团)
BrainNet-MoE: Brain-Inspired Mixture-of-Experts Learning for Neurological Disease Identification
The University of Texas at Arlington(德克萨斯大学阿灵顿分校);Indiana University Indianapolis(印第安纳大学印第安纳波利斯分校);University of Cambridge(剑桥大学);University of Sheffield(谢菲尔德大学);The University of Georgia(佐治亚大学)
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)数据科学与分析学域);Hong Kong Baptist University(香港浸会大学);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院);Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程学系)
Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
The University of Texas at Austin(德克萨斯大学奥斯汀分校);The University of Queensland(昆士兰大学);Univ. Grenoble Alpes(格勒诺布尔大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔理工学院);LJK(让·昆廷诺实验室)
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
Shanghai AI Laboratory(上海人工智能实验室);Beihang University(北京航空航天大学);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);University of Sydney(悉尼大学)
Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
Department of ECE The Ohio State University(电子工程系,俄亥俄州立大学);Department of CSE The Ohio State University(计算机科学与工程系,俄亥俄州立大学);Department of Computer Science University of Houston(计算机科学系,休斯顿大学);Department of ECE Department of CSE The Ohio State University(电子工程系、计算机科学与工程系,俄亥俄州立大学)