重新思考带有Sigmoid门函数的多项逻辑混合专家模型
Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
AI总结：
本文重新思考了带有Sigmoid门函数的多项逻辑混合专家模型，通过分析和改进门控函数，提高了模型的样本效率和收敛性。
摘要：
在混合专家（MoE）模型中，Sigmoid门在多个任务中已被实证证明在逼近前馈网络和语言建模等方面优于Softmax门。此外，最近的努力还表明，在回归设置下，Sigmoid门在样本效率上优于其Softmax对应物。然而，文献中尚有三个值得注意的问题未被解决，即（i）Sigmoid门在分类设置下的优势尚未确立；（ii）现有的Sigmoid门MoE模型可能无法收敛到其真实值；（iii）Sigmoid门中的温度参数的影响在理论上仍缺乏探索。为解决这些开放问题，我们对配备修改Sigmoid门的多项逻辑MoE进行了全面分析，以确保模型收敛。我们的结果表明，Sigmoid门在参数和专家估计方面均比Softmax门具有更低的样本复杂性。此外，我们发现在Sigmoid门中引入温度会导致样本复杂性呈指数级增长，这是由于温度与门控参数之间存在内在交互。为克服这一问题，我们提出将门控函数中的普通内积分数替换为欧几里得分数，从而有效去除该交互，从而大幅提高样本复杂性到多项式级。
英文摘要：
The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a polynomial order.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
