Autonomy-of-Experts 模型
Autonomy-of-Experts Models
AI总结:
针对MoE模型中路由器决策与专家执行分离导致的次优选择问题,提出AoE范式,通过专家基于内部激活范数自主选择并利用低秩分解降低开销,在700M至4B参数模型上证明其优于传统MoE。
摘要:
Mixture-of-Experts (MoE) 模型大多使用路由器将 token 分配给特定的专家模块,仅激活部分参数,通常优于稠密模型。我们认为,路由器的决策与专家的执行之间的分离是一个关键但被忽视的问题,导致了次优的专家选择和无效的学习。为了解决这个问题,我们提出了 Autonomy-of-Experts (AoE),一种新颖的 MoE 范式,其中专家自主选择自身来处理输入。AoE 基于这样一个洞察:专家能够感知自身有效处理 token 的能力,这种感知反映在其内部激活的规模上。在 AoE 中,路由器被移除;取而代之的是,专家为输入预计算内部激活,并根据其激活范数进行排名。只有排名靠前的专家继续进行前向传播,而其他专家则中止。预计算激活的开销通过低秩权重分解来降低。这种“自我评估后比较同伴”的方法确保了改进的专家选择和有效的学习。我们预训练了具有 700M 到 4B 参数的语言模型,证明 AoE 在具有可比效率的情况下优于传统的 MoE 模型。
英文摘要:
Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue that the separation between the router's decision-making and the experts' execution is a critical yet overlooked issue, leading to suboptimal expert selection and ineffective learning. To address this, we propose Autonomy-of-Experts (AoE), a novel MoE paradigm in which experts autonomously select themselves to process inputs. AoE is based on the insight that an expert is aware of its own capacity to effectively process a token, an awareness reflected in the scale of its internal activations. In AoE, routers are removed; instead, experts pre-compute internal activations for inputs and are ranked based on their activation norms. Only the top-ranking experts proceed with the forward pass, while the others abort. The overhead of pre-computing activations is reduced through a low-rank weight factorization. This self-evaluating-then-partner-comparing approach ensures improved expert selection and effective learning. We pre-train language models having 700M up to 4B parameters, demonstrating that AoE outperforms traditional MoE models with comparable efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。