关于层次混合专家中的专家估计：超越Softmax门控函数
On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
AI总结：
本文研究了层次混合专家（HMoE）模型，提出在模型两层结构中采用Laplace门控函数替代传统Softmax门控，以消除不良参数交互，从而加速专家收敛并增强专家专精化，实验验证了其在大规模多模态等任务上的显著性能提升。
摘要：
随着Mixture of Experts (MoE)架构在开发大规模基础模型中的重要性日益凸显，我们研究了Hierarchical Mixture of Experts (HMoE)，这是MoE的一种特殊变体，擅长处理复杂输入并提升特定任务上的性能。我们的分析强调了在HMoE框架中使用Laplace门控函数相较于传统Softmax门控的优势。我们在理论上证明，在HMoE模型的两层结构中应用Laplace门控函数有助于消除由Softmax门控引起的不良参数交互，从而加速专家收敛并增强专家的专精化。跨多种场景的实证验证支持了这些理论主张。这包括大规模多模态任务、图像分类以及潜在领域发现与预测任务，在这些任务中，我们修改后的HMoE模型与传统HMoE模型相比显示出巨大的性能提升。
英文摘要：
With the growing prominence of the Mixture of Experts (MoE) architecture in developing large-scale foundation models, we investigate the Hierarchical Mixture of Experts (HMoE), a specialized variant of MoE that excels in handling complex inputs and improving performance on targeted tasks. Our analysis highlights the advantages of using the Laplace gating function over the traditional Softmax gating within the HMoE frameworks. We theoretically demonstrate that applying the Laplace gating function at both levels of the HMoE model helps eliminate undesirable parameter interactions caused by the Softmax gating and, therefore, accelerates the expert convergence as well as enhances the expert specialization. Empirical validation across diverse scenarios supports these theoretical claims. This includes large-scale multimodal tasks, image classification, and latent domain discovery and prediction tasks, where our modified HMoE models show great performance improvements compared to the conventional HMoE models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
