专家分歧学习用于基于混合专家的语言模型
Expert Divergence Learning for MoE-based Language Models
AI总结：
专家分歧学习通过促进专家功能专业化，提升MoE模型的语言建模能力及下游任务性能。
摘要：
混合专家（MoE）架构是一种强大的技术，用于扩展语言模型，但往往受到专家同质化的影响，即专家学习冗余功能，从而限制了MoE的全部潜力。为了解决这一问题，我们引入了专家分歧学习，这是一种新的预训练策略，旨在明确鼓励专家之间的功能专业化。我们的方法结合了一个标签驱动的辅助损失，利用预训练语料库中固有的领域标签，以最大化不同数据域之间专家路由分布的Jensen-Shannon散度。这种优化目标引导模型为不同领域发展出分歧的路由策略，为相同领域发展出更接近的路由策略，从而产生出新的、有组织的专家专业化。我们通过从头开始预训练多达150亿参数的MoE模型来验证我们的方法。实验结果表明，使用专家分歧学习训练的模型不仅实现了更低的语言建模损失，还在各种下游基准测试中表现出显著的性能提升。进一步分析证实，我们的方法有效缓解了专家同质化，并带来了更大的功能专业化，所有这些在训练过程中都带来了极小的计算开销。
英文摘要：
The Mixture-of-Experts (MoE) architecture is a powerful technique for scaling language models, yet it often suffers from expert homogenization, where experts learn redundant functionalities, thereby limiting MoE's full potential. To address this, we introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts. Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora to maximize the Jensen-Shannon Divergence between the expert routing distributions of different data domains. This optimization objective guides the model to develop diverged routing policies for varied domains and closer routing policies for the same domain, which leads to emergent and organized expert specialization. We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch. Experimental results demonstrate that models trained with Expert Divergence Learning not only achieve a lower language modeling loss but also exhibit significant performance improvements across a diverse range of downstream benchmarks. Further analysis confirms that our method effectively mitigates expert homogenization and brings greater functional specialization, all with negligible computational overhead during training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
