利用压缩专家高效编辑混合专家模型
Efficiently Editing Mixture-of-Experts Models with Compressed Experts
AI总结:
针对MoE模型激活专家存在冗余的问题,提出用轻量压缩专家替代辅助激活专家的方法,可在保留90%以上性能的同时减少超30%激活参数、节省20%推理成本,助力资源受限场景部署。
摘要:
混合专家(Mixture-of-Experts, MoE)模型通过在训练和推理阶段仅激活部分专家,已成为高效扩展大语言模型的核心方法。通常而言,激活专家的数量存在权衡:专家数量越少,计算成本越低;专家数量越多,模型性能越好。近期研究表明,并非所有被激活的专家对模型性能的贡献都均等,部分专家的效用极低,尤其是在针对特定下游任务微调预训练MoE模型时更为明显。专家中重要参数与冗余参数的共存,为我们在维持模型性能的同时减少激活专家数量提供了契机。在本研究中,我们提出了压缩专家(compressed experts)的概念,即作为完整专家紧凑表示的轻量模块。我们的方法保留最重要的专家,同时用压缩专家替代其他辅助激活专家。激活参数的减少显著降低了推理成本,同时实现了与原模型相当的性能。在Phi-MoE、OLMoE等模型上开展的大量实验表明,压缩专家在各类任务上可恢复完整专家90%以上的性能,同时减少超过30%的激活参数,节省20%的推理成本。该方法支持MoE模型在资源受限场景下的高效部署,也有助于以可控的开销扩展到更大规模的模型。我们的代码可在https://github.com/yifei-he/Compressed-Experts获取。
英文摘要:
Mixture-of-Experts (MoE) models have become a key approach for scaling large language models efficiently by activating only a subset of experts during training and inference. Typically, the number of activated experts presents a trade-off: fewer experts reduce computational costs, while more experts improve performance. Recent studies reveal that not all activated experts contribute equally to model performance, with some providing minimal utility, particularly when finetuning pretrained MoE models for specialized downstream tasks. The co-existence of significant and redundant parameters in experts provides us an opportunity to reduce the number of activated experts while maintaining model performance. In this work, we propose the concept of compressed experts, lightweight modules that serve as compact representations of full experts. Our approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts. The reduction of active parameters significantly lowers inference costs while achieving comparable performance. Extensive experiments on models including Phi-MoE and OLMoE demonstrate that compressed experts recover over 90% of full expert performance across various tasks while reducing more than 30% active parameters and saving 20% in inference costs. This approach enables efficient deployment of MoE models in resource-constrained settings and facilitates scaling to larger models with manageable overhead. Our code is available at https://github.com/yifei-he/Compressed-Experts.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。