揭示专家内部激活稀疏性以实现高效的混合专家模型执行
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
AI总结：
本文研究了混合专家模型中专家内部激活稀疏性，通过利用这种未被探索的稀疏性维度，提高了模型执行效率，实现了2.5倍的层执行速度提升和1.2倍的端到端速度提升。
摘要：
混合专家（MoE）架构因其通过稀疏专家激活实现的计算效率而成为最先进的大型语言模型的标准。然而，由于专家坍塌和负载不平衡等基本训练挑战，通过更细粒度的专家粒度实现稀疏性变得越来越困难。在本文中，我们探索并利用专家内部激活稀疏性作为MoE模型稀疏性的一个互补且未被充分研究的维度。令人惊讶的是，现有预训练MoE模型中存在大量专家内部稀疏性，无需修改激活函数或模型参数，即可在每个专家内提供高达90%的稀疏性，而不会显著损失准确性。我们研究了八个现成的MoE模型（从1B到400B参数），并扩展了vLLM的MoE执行流水线，通过跳过非活动神经元的计算来利用专家内部激活稀疏性，在其现有优化基础上，实现了MoE层执行的2.5倍速度提升，以及与原始密集vLLM基线相比的1.2倍端到端速度提升。
英文摘要：
Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. However, sparsity through finer expert granularity is becoming increasingly difficult to achieve due to fundamental training challenges such as expert collapse and load imbalance. In this work, we explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models. Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models, without any modification to the activation function or model parameters, providing up to 90% sparsity within each expert without significant accuracy loss. We explore intra-expert activation sparsity across eight off-the-shelf MoE models ranging from 1B to 400B parameters, and extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons, on top of its existing optimizations, achieving up to 2.5 times speedup in MoE layer execution and 1.2 times end-to-end speedup compared to the original dense vLLM baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
