arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.06182cs.DCcs.ARcs.CLcs.LG
收藏

迈向MoE部署:缓解混合专家(MoE)推理中的低效问题

Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference

Meta AI;Duke University(杜克大学);University of Pennsylvania(宾夕法尼亚大学);Washington University in St. Louis(圣路易斯华盛顿大学)

Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S. Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, Benjamin Lee

AI总结:

针对MoE推理部署中的低效问题,提出动态门控、专家缓冲和负载均衡三种优化技术,显著提升吞吐量并降低内存占用。

摘要:

混合专家(MoE)模型在计算机视觉和自然语言处理的各种任务中取得了最先进的性能,因而广受欢迎。它们有效扩展了模型容量,同时在训练期间仅产生最小的计算成本增加。然而,由于此类模型规模庞大且通信模式复杂,将其部署用于推理十分困难。在本工作中,我们对两种MoE工作负载(即语言建模(LM)和机器翻译(MT))进行了特征描述,并识别了它们在部署时的低效来源。我们提出了三种优化技术以缓解低效来源,即(1)动态门控,(2)专家缓冲,以及(3)专家负载均衡。我们表明,动态门控将LM的最大吞吐量提高了6.21-11.23倍,MT编码器提高了5.75-10.98倍,MT解码器提高了2.58-5.71倍。它还将LM的内存使用量减少了最多1.36倍,MT减少了最多1.1倍。我们进一步提出了专家缓冲,一种新的缓存机制,仅将活跃的热专家保留在GPU内存中,而将其余专家缓冲在CPU内存中。这将静态内存分配减少了最多1.47倍。最后,我们提出了一种负载均衡方法,为工作负载提供了额外的可扩展性。

英文摘要:

Mixture-of-Experts (MoE) models have gained popularity in achieving state-of-the-art performance in a wide range of tasks in computer vision and natural language processing. They effectively expand the model capacity while incurring a minimal increase in computation cost during training. However, deploying such models for inference is difficult due to their large size and complex communication pattern. In this work, we provide a characterization of two MoE workloads, namely Language Modeling (LM) and Machine Translation (MT) and identify their sources of inefficiencies at deployment. We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing. We show that dynamic gating improves maximum throughput by 6.21-11.23$\times$ for LM, 5.75-10.98$\times$ for MT Encoder and 2.58-5.71$\times$ for MT Decoder. It also reduces memory usage by up to 1.36$\times$ for LM and up to 1.1$\times$ for MT. We further propose Expert Buffering, a new caching mechanism that only keeps hot, active experts in GPU memory while buffering the rest in CPU memory. This reduces static memory allocation by up to 1.47$\times$. We finally propose a load balancing methodology that provides additional scalability to the workload.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑