超越参数数量：软混合专家中的隐式偏置
Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
AI总结：
本文研究 Soft MoE 的隐式偏置，证明单个专家无法表示简单凸函数，说明多专家对表示能力是必要的，并提出一种在固定总参数量下高效近似专业化专家子集以减少推理计算的方法。
摘要：
关于稀疏混合专家（Sparse Mixture of Experts, MoE）模型的传统观点是：与其训练一个计算成本高昂的单一大型专家，不如训练许多小型专家。其期望是，如果小型专家的总参数数量与单一大型专家相等，那么我们就能在保留大型专家表示能力的同时，获得计算上的可处理性并促进专家专业化。最近提出的 Soft MoE 用可微的门控函数取代了 Sparse MoE 的离散路由机制，从而平滑地混合 token。虽然这种平滑门控函数成功缓解了与 Sparse MoE 相关的各种训练不稳定性，但尚不清楚它是否引入了影响 Soft MoE 表示能力或专家专业化潜力的隐式偏置。我们证明，具有单个任意强大专家的 Soft MoE 无法表示简单的凸函数。这说明了 Soft MoE 的成功不能用传统观点来解释，即许多小型专家集体模仿单一大型专家的表示能力；并且实际上需要多个专家才能获得良好的表示能力（即使在总参数数量固定的情况下）。沿着这一研究方向继续深入，我们为 Soft MoE 引入了一种专家专业化的概念，并在改变专家数量但固定总参数数量的条件下，考虑以下（计算上不可处理）任务：给定任意输入，我们如何发现专门用于预测该输入标签的专家子集？我们通过实验表明，当存在许多小型专家时，该架构会以一种隐式偏置的方式，使我们能够高效地近似出专业化的专家子集。我们的方法易于实现，并有望在推理过程中减少计算量。
英文摘要：
The traditional viewpoint on Sparse Mixture of Experts (MoE) models is that instead of training a single large expert, which is computationally expensive, we can train many small experts. The hope is that if the total parameter count of the small experts equals that of the singular large expert, then we retain the representation power of the large expert while gaining computational tractability and promoting expert specialization. The recently introduced Soft MoE replaces the Sparse MoE's discrete routing mechanism with a differentiable gating function that smoothly mixes tokens. While this smooth gating function successfully mitigates the various training instabilities associated with Sparse MoE, it is unclear whether it induces implicit biases that affect Soft MoE's representation power or potential for expert specialization. We prove that Soft MoE with a single arbitrarily powerful expert cannot represent simple convex functions. This justifies that Soft MoE's success cannot be explained by the traditional viewpoint of many small experts collectively mimicking the representation power of a single large expert, and that multiple experts are actually necessary to achieve good representation power (even for a fixed total parameter count). Continuing along this line of investigation, we introduce a notion of expert specialization for Soft MoE, and while varying the number of experts yet fixing the total parameter count, we consider the following (computationally intractable) task. Given any input, how can we discover the expert subset that is specialized to predict this input's label? We empirically show that when there are many small experts, the architecture is implicitly biased in a fashion that allows us to efficiently approximate the specialized expert subset. Our method can be easily implemented to potentially reduce computation during inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
