一种在微调稀疏混合专家模型中剪枝专家的可证明有效方法
A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
AI总结：
本文提出首个可证明高效的微调MoE模型专家剪枝方法，通过优先剪枝路由器l2范数变化较小的专家，在保证测试准确率的同时显著减少模型规模与计算需求，并在VMoE等大型视觉模型上验证有效。
摘要：
稀疏门控混合专家（MoE）架构通过可训练的路由器将不同的输入发送到不同的子网络，即专家。MoE显著减少了大型模型的训练计算量，但对于某些下游任务，其部署在内存或计算上可能仍然昂贵。模型剪枝是减少推理计算的常用方法，但其在MoE架构中的应用在很大程度上尚未被探索。据我们所知，本文提供了第一种可证明高效的微调MoE模型专家剪枝技术。我们从理论上证明，优先剪枝那些路由器l2范数相对于预训练模型变化较小的专家，能够保证保留测试准确率，同时显著减小模型规模和计算需求。尽管我们的理论分析集中在简化MoE架构上的二元分类任务，但我们的专家剪枝方法在大型视觉MoE模型（如VMoE和E3MoE）上得到了验证，这些模型在CIFAR10、CIFAR100和ImageNet等基准数据集上进行了微调。
英文摘要：
The sparsely gated mixture of experts (MoE) architecture sends different inputs to different subnetworks, i.e., experts, through trainable routers. MoE reduces the training computation significantly for large models, but its deployment can be still memory or computation expensive for some downstream tasks. Model pruning is a popular approach to reduce inference computation, but its application in MoE architecture is largely unexplored. To the best of our knowledge, this paper provides the first provably efficient technique for pruning experts in finetuned MoE models. We theoretically prove that prioritizing the pruning of the experts with a smaller change of the routers l2 norm from the pretrained model guarantees the preservation of test accuracy, while significantly reducing the model size and the computational requirements. Although our theoretical analysis is centered on binary classification tasks on simplified MoE architecture, our expert pruning method is verified on large vision MoE models such as VMoE and E3MoE finetuned on benchmark datasets such as CIFAR10, CIFAR100, and ImageNet.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
