arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.02500cs.LGcs.AI
收藏

迈向高效的混合专家模型:压缩技术的整体性研究

Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques

University of Maryland, College Park(马里兰大学帕克分校);Rutgers University(罗格斯大学);University of Sydney(悉尼大学)

Shwai He, Daize Dong, Liang Ding, Ang Li

AI总结:

本文系统研究MoE压缩,提出Layer Drop、Block Drop并结合Expert Slimming与Expert Trimming,在Mixtral-8x7B上以极小性能损失实现6.05倍加速和77.1%内存削减。

摘要:

扩展大型语言模型已推动多个领域取得显著进展,但模型规模的持续增长给实际部署带来了重大挑战。混合专家(Mixture of Experts,MoE)架构提供了一种有前景的解决方案:在推理过程中仅动态选择并激活一部分专家,从而在保持高性能的同时大幅降低计算成本。尽管具备这些优势,MoE也引入了新的低效问题,例如参数过多和通信开销。在本研究中,我们对混合专家的压缩技术进行了整体性研究,以提升效率和可扩展性。近期的研究主要集中在Expert Trimming(专家裁剪)上,即减少专家数量,但这些方法仍承受相当大的通信和计算成本。为解决这一问题,我们提出了更激进的策略,例如移除整个MoE层的Layer Drop,以及消除transformer块的Block Drop。令人惊讶的是,这些激进的剪枝技术不仅保持了模型性能,还显著提升了计算和内存效率。此外,除Expert Trimming之外,我们还引入了Expert Slimming(专家瘦身),用于压缩单个专家以进一步提升性能,并且能够与Expert Trimming无缝集成。大量实验结果证明了我们提出的Layer Drop和Block Drop方法的有效性,以及整合Expert Slimming与Expert Trimming的综合方案的有效性:在Mixtral-8x7B上实现了6.05倍加速,内存使用减少77.1%,同时保持超过92%的性能。我们的代码已发布于https://github.com/CASE-Lab-UMD/Unified-MoE-Compression。

英文摘要:

Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world deployment. The Mixture of Experts (MoE) architecture offers a promising solution by dynamically selecting and activating only a subset of experts during inference, thus substantially reducing computational costs while preserving high performance. Despite these benefits, MoE introduces new inefficiencies, such as excessive parameters and communication overhead. In this work, we present a holistic study of compression techniques for Mixture of Experts to enhance both efficiency and scalability. While recent efforts have focused on Expert Trimming, which reduces the number of experts, these approaches still suffer from considerable communication and computational costs. To address this, we propose more aggressive strategies, such as Layer Drop, which removes entire MoE layers, and Block Drop, which eliminates transformer blocks. Surprisingly, these aggressive pruning techniques not only preserve model performance but also substantially improve computation and memory efficiency. Furthermore, beyond Expert Trimming, we also introduce Expert Slimming, which compresses individual experts to further boost performance and can be seamlessly integrated with Expert Trimming. Extensive experimental results demonstrate the effectiveness of our proposed methods-Layer Drop and Block Drop-along with the comprehensive recipe that integrates Expert Slimming and Expert Trimming, achieving a 6.05x speedup with 77.1% reduced memory usage while maintaining over 92% of performance on Mixtral-8x7B. Our code is released at https://github.com/CASE-Lab-UMD/Unified-MoE-Compression.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑