每位专家都很重要：面向混合专家语言模型的有效知识蒸馏
Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
AI总结：
针对MoE语言模型压缩中未激活专家知识未被有效利用的问题，提出KA和SAR两种专用KD方法，通过采样增强和路由器训练提取所有专家知识，实验证明其优于传统KD方法。
摘要：
随着混合专家模型（Mixture-of-Experts, MoE）的出现，模型规模的高效扩展近年来加速了大语言模型的发展。然而，其高内存需求使其无法在资源受限环境中使用。尽管知识蒸馏（KD）是一种成熟的模型压缩方法，但其在MoE教师模型上的应用仍未得到充分探索。通过研究，我们发现MoE模型中未激活的专家拥有对学生模型有益的宝贵知识。我们进一步证明，现有KD方法并非压缩MoE模型的最佳选择，因为它们未能有效利用这些知识。为此，我们首次提出两种直观的MoE专用KD方法：知识增强（KA）和学生感知路由器（SAR），两者均旨在有效提取所有专家的知识。具体而言，KA通过多次采样专家来增强知识，而SAR使用所有专家并通过路由器训练调整专家权重以提供最优知识。大量实验表明，我们的方法优于传统KD方法，证明了其对MoE教师模型的有效性。
英文摘要：
With the emergence of Mixture-of-Experts (MoE), the efficient scaling of model size has accelerated the development of large language models in recent years. However, their high memory requirements prevent their use in resource-constrained environments. While knowledge distillation (KD) has been a proven method for model compression, its application to MoE teacher models remains underexplored. Through our investigation, we discover that non-activated experts in MoE models possess valuable knowledge that benefits student models. We further demonstrate that existing KD methods are not optimal for compressing MoE models, as they fail to leverage this knowledge effectively. To address this, we propose two intuitive MoE-specific KD methods for the first time: Knowledge Augmentation (KA) and Student-Aware Router (SAR), both designed to effectively extract knowledge from all experts. Specifically, KA augments knowledge by sampling experts multiple times, while SAR uses all experts and adjusts the expert weights through router training to provide optimal knowledge. Extensive experiments show that our methods outperform conventional KD methods, demonstrating their effectiveness for MoE teacher models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
