Mixture-of-Experts (MoE) models have gained popularity in achieving state-of-the-art performance in a wide range of tasks in computer vision and natural language processing. They effectively expand the model capacity while incurring a minimal increase in computation cost during training. However, deploying such models for inference is difficult due to their large size and complex communication pattern. In this work, we provide a characterization of two MoE workloads, namely Language Modeling (LM) and Machine Translation (MT) and identify their sources of inefficiencies at deployment. We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing. We show that dynamic gating improves maximum throughput by 6.21-11.23$\times$ for LM, 5.75-10.98$\times$ for MT Encoder and 2.58-5.71$\times$ for MT Decoder. It also reduces memory usage by up to 1.36$\times$ for LM and up to 1.1$\times$ for MT. We further propose Expert Buffering, a new caching mechanism that only keeps hot, active experts in GPU memory while buffering the rest in CPU memory. This reduces static memory allocation by up to 1.47$\times$. We finally propose a load balancing methodology that provides additional scalability to the workload.
SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(人工智能国家重点实验室,智能科学与技术学院,北京大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
Institute of Automation, CAS(中国科学院自动化研究所);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院);Shanghai Jiao Tong University(上海交通大学);AiRiA
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
Zhejiang University(浙江大学);Shanghai Innovation Institute(上海创新研究院);ByteDance Inc.(字节跳动公司);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Department of Automation, Tsinghua University(自动化系,清华大学);Academy of Medical Engineering and Translational Medicine, Tianjin University(医学工程与转化医学学院,天津大学)
Merging Multi-Task Models via Weight-Ensembling Mixture of Experts
National Engineering Research Center for Multimedia Software(国家多媒体软件工程技术研究中心);School of Computer Science, Wuhan University(武汉大学计算机学院);Hubei Luojia Laboratory(湖北珞珈实验室);Sun Yat-sen University(中山大学);JD Explore Academy(京东探索研究院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Nanyang Technological University(南洋理工大学)
One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning
Trivita AI;The University of Texas at Austin(德克萨斯大学奥斯汀分校);Northeastern University(东北大学);Hanoi University of Science and Technology(河内科学技术大学);Rutgers University(罗格斯大学)