MoE-I$^2$：通过专家间剪枝与专家内低秩分解压缩混合专家模型
MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
AI总结：
针对MoE大模型参数量大、部署成本高的问题，提出结合专家间非均匀剪枝与专家内低秩分解的两阶段压缩方法，在多个主流MoE模型上验证了其可在保持零样本任务性能的同时压缩模型、提升推理效率。
摘要：
混合专家（Mixture of Experts, MoE）大语言模型（LLM）的出现极大推动了语言模型的发展。与传统LLM相比，MoE LLM以显著更少的激活参数实现了更高的性能，表现更优。尽管具备这种效率，其庞大的参数量仍导致部署成本高昂。本文提出一种专为MoE设计的两阶段压缩方法，以减小模型规模并降低计算成本。首先，在专家间剪枝阶段，我们分析了每一层的重要性，提出采用非均匀剪枝率的逐层遗传搜索与分块KT感受野方法，对单个专家进行剪枝。其次，在专家内分解阶段，我们应用低秩分解进一步压缩剩余专家内部的参数。在Qwen1.5-MoE-A2.7B、DeepSeek-V2-Lite和Mixtral-8×7B上开展的大量实验表明，我们提出的方法既能减小模型规模、提升推理效率，同时还能在各类零样本任务中保持性能。代码将发布于https://github.com/xiaochengsky/MoEI-2.git。
英文摘要：
The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by achieving higher performance with considerably fewer activated parameters. Despite this efficiency, their enormous parameter size still leads to high deployment costs. In this paper, we introduce a two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost. First, in the inter-expert pruning stage, we analyze the importance of each layer and propose the Layer-wise Genetic Search and Block-wise KT-Reception Field with the non-uniform pruning ratio to prune the individual expert. Second, in the intra-expert decomposition stage, we apply the low-rank decomposition to further compress the parameters within the remaining experts. Extensive experiments on Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Mixtral-8$\times$7B demonstrate that our proposed methods can both reduce the model size and enhance inference efficiency while maintaining performance in various zero-shot tasks. The code will be available at \url{https://github.com/xiaochengsky/MoEI-2.git}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
