视觉混合专家中的路由器：一项实证研究
Routers in Vision Mixture of Experts: An Empirical Study
AI总结：
本文实证研究了视觉混合专家模型中的路由器，提出统一公式并对比六种路由器，发现专家选择优于令牌选择，软MoE优于稀疏MoE。
摘要：
混合专家（MoE）模型是一种在不显著增加计算成本的情况下扩展模型容量的有前景的方法。MoE的一个关键组件是路由器，它决定哪些参数子集（专家）处理哪些特征嵌入（令牌）。在本文中，我们对计算机视觉任务中的MoE路由器进行了全面研究。我们引入了一个统一的MoE公式，该公式通过两个参数化路由张量涵盖了不同的MoE。该公式既包括稀疏MoE（在专家和令牌之间使用二元或硬分配），也包括软MoE（在专家和令牌的加权组合之间使用软分配）。稀疏MoE的路由器可进一步分为两种变体：令牌选择（将专家匹配到每个令牌）和专家选择（将令牌匹配到每个专家）。我们使用6种不同的路由器进行了头对头实验，包括先前工作中的现有路由器和我们引入的新路由器。我们表明：（i）许多最初为语言建模开发的路由器可以适应并在视觉任务中表现强劲，（ii）在稀疏MoE中，专家选择路由器通常优于令牌选择路由器，（iii）在固定计算预算下，软MoE通常优于稀疏MoE。这些结果为路由器在视觉MoE模型中的关键作用提供了新的见解。
英文摘要：
Mixture-of-Experts (MoE) models are a promising way to scale up model capacity without significantly increasing computational cost. A key component of MoEs is the router, which decides which subset of parameters (experts) process which feature embeddings (tokens). In this paper, we present a comprehensive study of routers in MoEs for computer vision tasks. We introduce a unified MoE formulation that subsumes different MoEs with two parametric routing tensors. This formulation covers both sparse MoE, which uses a binary or hard assignment between experts and tokens, and soft MoE, which uses a soft assignment between experts and weighted combinations of tokens. Routers for sparse MoEs can be further grouped into two variants: Token Choice, which matches experts to each token, and Expert Choice, which matches tokens to each expert. We conduct head-to-head experiments with 6 different routers, including existing routers from prior work and new ones we introduce. We show that (i) many routers originally developed for language modeling can be adapted to perform strongly in vision tasks, (ii) in sparse MoE, Expert Choice routers generally outperform Token Choice routers, and (iii) soft MoEs generally outperform sparse MoEs with a fixed compute budget. These results provide new insights regarding the crucial role of routers in vision MoE models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
