自适应倒排索引路由用于细粒度混合专家模型
Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
AI总结：
针对细粒度MoE模型路由开销大的问题，提出基于向量量化的两阶段倒排索引路由AIR-MoE，近似Top-K选择，无需结构约束，在百万级专家下改善困惑度-FLOPs权衡。
摘要：
稀疏混合专家（MoE）模型通过每个令牌仅激活一部分专家来实现可扩展的Transformer架构。近期证据表明，随着专家粒度越来越细，即使用许多小专家而非少数大专家，模型性能会得到提升。然而，这种模式大幅增加了路由成本，可能主导计算和内存开销。我们提出了用于MoE的自适应倒排索引路由（AIR-MoE），一种基于向量量化（VQ）的倒排索引启发的路由架构。在第一阶段，AIR-MoE通过将令牌分配到VQ码字来构建专家候选集，从而进行粗略的初选。在第二阶段，精细评分仅针对该候选集计算精确的路由分数。这种两阶段过程近似于真正的Top-K路由，同时避免了对所有专家进行完整评分，并且与先前工作不同，不对专家参数施加任何结构约束。AIR-MoE可作为标准路由器的即插即用替代方案，无需修改模型架构或损失函数。我们进一步提供了AIR-MoE所实现的质量召回率的下界，这有助于理解其内部工作机制。实验上，在拥有超过一百万个微小专家的场景中，AIR-MoE相较于现有MoE路由器改善了困惑度-浮点运算次数（FLOPs）的权衡，并在与最佳现有基线的比较中实现了持续的困惑度改进。
英文摘要：
Sparse Mixture-of-experts (MoE) models enable scalable transformer architectures by activating only a subset of experts per token. Recent evidence suggests that performance improves with increasingly granular experts, i.e., many small experts instead of a few large ones. However, this regime substantially increases routing cost, which can dominate computation and memory. We introduce adaptive inverted-index routing for MoE (AIR-MoE), an inverted-index-inspired routing architecture based on vector quantization (VQ). In a first stage, AIR-MoE performs coarse shortlisting by assigning tokens to VQ codewords to construct a candidate set of experts. In a second stage, fine scoring computes exact routing scores restricted to this shortlist. This two-stage procedure approximates true top-K routing while avoiding full expert scoring and, in contrast to prior work, imposing no structural constraints on expert parameters. AIR-MoE serves as a drop-in replacement for standard routers and requires no modifications to the model architecture or loss function. We further provide a lower bound on the mass recall achieved by AIR-MoE that yields insights into the inner workings. Empirically, AIR-MoE improves the perplexity-FLOPs trade-off over existing MoE routers in regimes with up to more than a million tiny experts, and achieves consistent perplexity improvements over the best existing baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
