MoDE：一种专家间具备相互蒸馏机制的混合专家模型
MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts
AI总结：
针对混合专家（MoE）模型因门控路由导致专家视野狭窄、泛化能力受限的问题，提出专家间相互蒸馏的MoDE模型，经多领域实验验证有效，并通过专家探测揭示其作用机理。
摘要：
混合专家（MoE）模型因具备提升模型性能的能力，应用正日益广泛。在MoE结构中，门控层在区分输入特征并将其路由至不同专家的过程中发挥着关键作用，这使得每位专家都能专门处理对应的子任务。然而，门控的路由机制也会引发视野狭窄的问题：单个MoE专家无法利用更多样本学习所分配的子任务，这反过来限制了MoE进一步提升泛化能力。为有效解决该问题，我们提出了名为Mixture-of-Distilled-Expert（MoDE，蒸馏专家混合模型）的方法，该方法在专家之间施加适度的相互蒸馏，使每位专家都能习得更多其他专家学到的特征，并对自身最初分配的子任务形成更准确的认知。我们在表格数据集、自然语言处理（NLP）数据集和计算机视觉（CV）数据集上开展了大量实验，结果表明MoDE具备有效性、通用性和鲁棒性。此外，我们通过创新性地构建“专家探测”开展了一项平行研究，从实验层面证明了MoDE的生效机理：适度的知识蒸馏能够提升每位专家在所分配任务上的测试性能，进而带来MoE整体性能的提升。
英文摘要：
The application of mixture-of-experts (MoE) is gaining popularity due to its ability to improve model's performance. In an MoE structure, the gate layer plays a significant role in distinguishing and routing input features to different experts. This enables each expert to specialize in processing their corresponding sub-tasks. However, the gate's routing mechanism also gives rise to narrow vision: the individual MoE's expert fails to use more samples in learning the allocated sub-task, which in turn limits the MoE to further improve its generalization ability. To effectively address this, we propose a method called Mixture-of-Distilled-Expert (MoDE), which applies moderate mutual distillation among experts to enable each expert to pick up more features learned by other experts and gain more accurate perceptions on their original allocated sub-tasks. We conduct plenty experiments including tabular, NLP and CV datasets, which shows MoDE's effectiveness, universality and robustness. Furthermore, we develop a parallel study through innovatively constructing "expert probing", to experimentally prove why MoDE works: moderate distilling knowledge can improve each individual expert's test performances on their assigned tasks, leading to MoE's overall performance improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
