Large language models have transformed many applications but remain expensive to train. Sparse Mixture of Experts (MoE) addresses this through conditional computation, with Expert Parallel (EP) as the standard distributed training method. However, EP has three limitations: communication cost grows linearly with the number of activated experts $k$, load imbalance affects latency and memory usage, and data-dependent communication requires metadata exchange. We propose Multi-Head LatentMoE and Head Parallel (HP), a new architecture and parallelism achieving $O(1)$ communication cost regardless of $k$, completely balanced traffic, and deterministic communication, all while remaining compatible with EP. To accelerate Multi-Head LatentMoE, we propose IO-aware routing and expert computation. Compared to MoE with EP, Multi-Head LatentMoE with HP trains up to $1.61\times$ faster while having identical performance. With doubled granularity, it achieves higher overall performance while still being $1.11\times$ faster. Our method makes multi-billion-parameter foundation model research more accessible.
University of Southern California（南加州大学）；AWS AI（亚马逊云科技人工智能部门）
The mixture of Expert (MoE) parallelism is a recent advancement that scales up the model size with constant computational cost. MoE selects different sets of parameters (i.e., experts) for each incoming token, resulting in a sparsely-activated model. Despite several successful applications of MoE, its training efficiency degrades significantly as the number of experts increases. The routing stage in MoE relies on the efficiency of the All2All communication collective, which suffers from network congestion and has poor scalability. To mitigate these issues, we introduce SMILE, which exploits he…
FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）数据科学与分析学域）；Hong Kong Baptist University（香港浸会大学）；School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程学系）
The Ohio State University（俄亥俄州立大学）；Microsoft Corporation（微软公司）
In recent years, the training requirements of many state-of-the-art Deep Learning (DL) models have scaled beyond the compute and memory capabilities of a single processor, and necessitated distribution among processors. Training such massive models necessitates advanced parallelism strategies to maintain efficiency. However, such distributed DL parallelism strategies require a varied mixture of collective and point-to-point communication operations across a broad range of message sizes and scales. Examples of models using advanced parallelism strategies include Deep Learning Recommendation Mod…
Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Minnesota Twin Cities（明尼苏达大学双城分校）；The Chinese University of Hong Kong（香港中文大学）
Beihang University（北京航空航天大学）；The Institute of Acoustics of the Chinese Academy of Sciences（中国科学院声学研究所）；Microsoft（微软）
The sparse Mixture-of-Experts (MoE) model is powerful for large-scale pre-training and has achieved promising results due to its model capacity. However, with trillions of parameters, MoE is hard to be deployed on cloud or mobile environment. The inference of MoE requires expert parallelism, which is not hardware-friendly and communication expensive. Especially for resource-limited downstream tasks, such sparse structure has to sacrifice a lot of computing efficiency for limited performance gains. In this work, we observe most experts contribute scarcely little to the MoE fine-tuning and infer…
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Tongji University（同济大学）；Cornell University（康奈尔大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；AI Training Platform Team, Shenzhen Loop Area Institute（深圳环域研究院AI训练平台团队）
The Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast with dense models, the sparse architecture of MoE offers opportunities for drastically growing model size with significant accuracy gain while consuming much lower compute budget. However, supporting large scale MoE training also has its own set of system and modeling challenges. To overcome the challenges and embrace the opportunities of MoE, we first develop a system capable of scaling MoE models efficiently to tr…
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
School of Software, Tsinghua University, Beijing, China（清华大学软件学院，北京，中国）；TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China（天津大学计算机科学与技术学院，中国）；School of New Media and Communication, Tianjin University, China（天津大学新媒体与传播学院，中国）
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
University of Utah（犹他大学）；University of Waterloo（滑铁卢大学）；The University of Queensland（昆士兰大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Virginia（弗吉尼亚大学）
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
Tsinghua University（清华大学）；Xingyun Integrated Circuits Co., Ltd.（星云集成电路有限公司）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
