从稀疏到软混合专家
From Sparse to Soft Mixtures of Experts
AI总结：
本文提出Soft MoE，一种完全可微分的稀疏Transformer，通过隐式软分配解决传统MoE的训练不稳定、令牌丢弃等问题，在视觉识别中大幅超越密集Transformer和现有MoE，并以极小推理开销实现参数规模大幅扩展。
摘要：
稀疏混合专家架构（MoEs）在不显著增加训练或推理成本的情况下扩展了模型容量。尽管取得了成功，MoEs仍存在一系列问题：训练不稳定、令牌丢弃、无法扩展专家数量以及微调效果不佳。在这项工作中，我们提出了Soft MoE，一种完全可微分的稀疏Transformer，它解决了这些挑战，同时保留了MoEs的优势。Soft MoE通过将不同加权组合的所有输入令牌传递给每个专家，执行隐式软分配。与其他MoE一样，Soft MoE中的专家仅处理（组合后的）令牌的子集，从而在较低的推理成本下实现更大的模型容量（和性能）。在视觉识别领域，Soft MoE大幅优于密集Transformer（ViTs）和流行的MoEs（Tokens Choice和Experts Choice）。此外，Soft MoE扩展性良好：具有128个专家分布在16个MoE层的Soft MoE Huge/14，其参数比ViT Huge/14多40倍以上，而推理时间仅增加2%，且质量显著提高。
英文摘要：
Sparse mixture of expert architectures (MoEs) scale model capacity without significant increases in training or inference costs. Despite their success, MoEs suffer from a number of issues: training instability, token dropping, inability to scale the number of experts, or ineffective finetuning. In this work, we propose Soft MoE, a fully-differentiable sparse Transformer that addresses these challenges, while maintaining the benefits of MoEs. Soft MoE performs an implicit soft assignment by passing different weighted combinations of all input tokens to each expert. As in other MoEs, experts in Soft MoE only process a subset of the (combined) tokens, enabling larger model capacity (and performance) at lower inference cost. In the context of visual recognition, Soft MoE greatly outperforms dense Transformers (ViTs) and popular MoEs (Tokens Choice and Experts Choice). Furthermore, Soft MoE scales well: Soft MoE Huge/14 with 128 experts in 16 MoE layers has over 40x more parameters than ViT Huge/14, with only 2% increased inference time, and substantially better quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
