Pre-gated MoE：面向快速且可扩展的混合专家推理的算法-系统协同设计
Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
AI总结：
本文提出Pre-gated MoE算法-系统协同设计，通过新颖的预门控函数缓解稀疏专家动态激活问题，在保持模型质量的同时降低GPU内存消耗并提升性能，实现单GPU高效低成本部署大规模LLMs。
摘要：
近年来，基于transformer的大型语言模型（LLMs）取得了重大进展，其成功得益于模型规模的扩大。尽管LLMs具有很高的算法性能，但其计算和内存需求带来了前所未有的挑战。为了应对LLMs的高计算需求，人们引入了混合专家架构，该架构能够在不按比例增加计算需求的情况下扩大模型规模。然而，MoE的高内存需求和稀疏专家的动态激活限制了其在实际问题中的适用性。以往将MoE消耗大量内存的专家参数卸载到CPU内存的解决方案存在不足，因为将激活的专家从CPU迁移到GPU的延迟会导致很高的性能开销。我们提出的Pre-gated MoE系统利用我们的算法-系统协同设计，有效解决了传统MoE架构的计算和内存挑战。Pre-gated MoE采用了我们新颖的预门控函数，减轻了稀疏专家激活的动态特性，使我们提出的系统能够解决MoE庞大的内存占用问题，同时实现高性能。我们证明，Pre-gated MoE能够提升性能、降低GPU内存消耗，同时保持同等水平的模型质量。这些特性使得我们的Pre-gated MoE系统能够仅使用单个GPU，以高性能且高性价比地部署大规模LLMs。
英文摘要：
Large language models (LLMs) based on transformers have made significant strides in recent years, the success of which is driven by scaling up their model size. Despite their high algorithmic performance, the computational and memory requirements of LLMs present unprecedented challenges. To tackle the high compute requirements of LLMs, the Mixture-of-Experts (MoE) architecture was introduced which is able to scale its model size without proportionally scaling up its computational requirements. Unfortunately, MoE's high memory demands and dynamic activation of sparse experts restrict its applicability to real-world problems. Previous solutions that offload MoE's memory-hungry expert parameters to CPU memory fall short because the latency to migrate activated experts from CPU to GPU incurs high performance overhead. Our proposed Pre-gated MoE system effectively tackles the compute and memory challenges of conventional MoE architectures using our algorithm-system co-design. Pre-gated MoE employs our novel pre-gating function which alleviates the dynamic nature of sparse expert activation, allowing our proposed system to address the large memory footprint of MoEs while also achieving high performance. We demonstrate that Pre-gated MoE is able to improve performance, reduce GPU memory consumption, while also maintaining the same level of model quality. These features allow our Pre-gated MoE system to cost-effectively deploy large-scale LLMs using just a single GPU with high performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
