协同的内层和跨层正则化损失用于MoE专家专业化
Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
AI总结：
本文提出两种正则化损失，通过增强MoE的专业化和路由效率，提升模型性能和推理速度。
摘要：
稀疏混合专家（MoE）模型能够高效地扩展Transformer，但存在专家重叠的问题——即专家之间存在冗余的表示和路由模糊，导致模型容量严重未被充分利用。尽管像DeepSeekMoE这样的架构解决方案可以促进专业化，但它们需要大量的结构修改，并且仅依赖于同一层的信号。在本文中，我们提出两种即插即用的正则化损失，以增强MoE的专业化和路由效率，而无需修改路由或模型架构。首先，一个内层专业化损失会惩罚相同token上专家的SwiGLU激活之间的余弦相似性，促使专家在互补的知识上专业化。其次，一个跨层耦合损失会最大化相邻层之间的联合Top-k路由概率，通过网络深度建立连贯的专家路径，同时强化同一层专家的专业化。这两种损失与标准负载平衡损失正交，并且适用于DeepSeekMoE中的共享专家架构和普通的top-k MoE架构。我们将这两种损失作为Megatron-LM模块实现。在预训练、微调和零样本基准上的广泛实验表明，一致的任务增益、更高的专家专业化程度和更低熵的路由；这些改进共同转化为更稳定的专家路径，从而加快推理速度。
英文摘要：
Sparse Mixture-of-Experts (MoE) models scale Transformers efficiently but suffer from expert overlap -- redundant representations across experts and routing ambiguity, resulting in severely underutilized model capacity. While architectural solutions like DeepSeekMoE promote specialization, they require substantial structural modifications and rely solely on intra-layer signals. In this paper, we propose two plug-and-play regularization losses that enhance MoE specialization and routing efficiency without modifying router or model architectures. First, an intra-layer specialization loss penalizes cosine similarity between experts' SwiGLU activations on identical tokens, encouraging experts to specialize in complementary knowledge. Second, a cross-layer coupling loss maximizes joint Top-$k$ routing probabilities across adjacent layers, establishing coherent expert pathways through network depth while reinforcing intra-layer expert specialization. Both losses are orthogonal to the standard load-balancing loss and compatible with both the shared-expert architecture in DeepSeekMoE and vanilla top-$k$ MoE architectures. We implement both losses as a drop-in Megatron-LM module. Extensive experiments across pre-training, fine-tuning, and zero-shot benchmarks demonstrate consistent task gains, higher expert specialization, and lower-entropy routing; together, these improvements translate into faster inference via more stable expert pathways.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
