PuzzleMoE：通过稀疏专家合并和位打包推理对大型专家混合模型进行高效压缩
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
AI总结：
研究针对专家混合模型内存开销大难题，提出PuzzleMoE方法。通过识别权重冗余与专业化进行稀疏专家合并，用双掩码捕获参数，引入位打包编码避免存储开销，实现高效推理，大幅压缩模型且保持精度。
摘要：
专家混合（MoE）模型在有效扩展语言模型方面潜力巨大，但因存储所有专家参数的高内存开销，其广泛部署受限。本文介绍PuzzleMoE，一种无需训练的MoE压缩方法，通过两项关键创新实现高精度和高效推理。实验表明，PuzzleMoE可将MoE模型压缩高达50%，同时保持精度。
英文摘要：
Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input. However, their widespread deployment remains limited due to the high memory overhead associated with storing all expert parameters, particularly as the number of experts increases. To address this challenge, prior works have explored expert dropping and merging strategies, yet they often suffer from performance drop at high compression ratios. In this paper, we introduce PuzzleMoE, a training-free MoE compression method that achieves both high accuracy and efficient inference through two key innovations: First, PuzzleMoE performs sparse expert merging by identifying element-wise weight redundancy and specialization. It uses a dual-mask to capture both shared and expert-specific parameters. Second, to avoid the overhead of storing binary masks and signs, PuzzleMoE introduces a bit-packed encoding scheme that reuses underutilized exponent bits, enabling efficient MoE inference on GPUs. Extensive experiments demonstrate that PuzzleMoE can compress MoE models by up to 50% while maintaining accuracy across various tasks. Specifically, it outperforms prior MoE compression methods by up to 16.7% on MMLU at 50% compression ratio, and achieves up to 1.28\times inference speedup.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
