LightMoE: 通过专家替换减少混合专家冗余
LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
AI总结：
本文提出LightMoE框架，通过专家替换、自适应选择和分层构造提升混合专家模型的内存效率和性能，实验表明其在30%压缩下性能与LoRA相当，50%压缩下表现更优。
摘要：
基于混合专家（MoE）的大型语言模型（LLMs）展现了出色的性能和计算效率，但其部署常受限于巨大的内存需求，主要由于需加载大量专家模块。现有专家压缩技术如剪枝或合并常导致不可逆的知识损失或高训练开销。本文提出一种新颖的专家压缩范式——专家替换，通过用参数高效模块替换冗余专家并以低成本恢复其能力。我们发现，即使该范式的基础基线也表现出色。在此基础上，我们引入LightMoE框架，通过自适应专家选择、分层专家构造和退火恢复策略增强该范式。实验结果表明，LightMoE在30%压缩率下与LoRA微调性能相当，即使在更激进的50%压缩率下，其在五个多样化任务上平均性能提升5.6%。这些发现表明，LightMoE在内存效率、训练效率和模型性能之间实现了更优的平衡。
英文摘要：
Mixture-of-Experts (MoE) based Large Language Models (LLMs) have demonstrated impressive performance and computational efficiency. However, their deployment is often constrained by substantial memory demands, primarily due to the need to load numerous expert modules. While existing expert compression techniques like pruning or merging attempt to mitigate this, they often suffer from irreversible knowledge loss or high training overhead. In this paper, we propose a novel expert compression paradigm termed expert replacing, which replaces redundant experts with parameter-efficient modules and recovers their capabilities with low training costs. We find that even a straightforward baseline of this paradigm yields promising performance. Building on this foundation, we introduce LightMoE, a framework that enhances the paradigm by introducing adaptive expert selection, hierarchical expert construction, and an annealed recovery strategy. Experimental results show that LightMoE matches the performance of LoRA fine-tuning at a 30% compression ratio. Even under a more aggressive 50% compression rate, it outperforms existing methods and achieves average performance improvements of 5.6% across five diverse tasks. These findings demonstrate that LightMoE strikes a superior balance among memory efficiency, training efficiency, and model performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
