Tutel：大规模自适应混合专家模型
Tutel: Adaptive Mixture-of-Experts at Scale
AI总结：
针对MoE动态令牌路由导致静态并行与流水线效率不足的问题，本文提出Flex技术栈，通过统一参数与数据布局实现零成本运行时自适应，并在大规模GPU和真实视觉模型上验证显著加速与效果。
摘要：
稀疏门控混合专家模型（MoE）已被广泛用于在固定计算成本下将深度学习模型扩展至万亿以上参数。MoE的算法性能依赖于其令牌路由机制，该机制将每个输入令牌转发给合适的子模型或专家。尽管令牌路由会在运行时动态决定专家工作量，但现有系统由于采用静态执行方式，即静态并行和流水线，无法适应动态工作负载，因而存在计算效率低下的问题。我们提出了Flex，这是一种面向MoE的高度可扩展技术栈设计与实现，具备动态自适应并行和流水线能力。Flex为分发MoE模型参数和输入数据设计了统一布局，所有可能的并行或流水线方法都可以利用该布局，而不会产生任何数学不等价或张量迁移开销。这使得运行时能够以零成本进行自适应并行/流水线优化。基于这一关键设计，Flex还实现了多种MoE加速技术。综合所有技术，Flex最终在任意规模下都带来了显著加速：与此前最先进方法相比，单个MoE层在16块和2,048块A100 GPU上分别实现4.96倍和5.75倍加速。我们的评估表明，Flex能够高效且有效地运行名为SwinV2-MoE的真实MoE模型，该模型构建于最先进的计算机视觉架构Swin Transformer V2之上。在效率方面，Flex加速了SwinV2-MoE，与Fairseq相比，训练和推理分别实现最高1.55倍和2.11倍加速。在有效性方面，SwinV2-MoE模型在预训练以及COCO目标检测等下游计算机视觉任务中均取得了优于对应密集模型的准确率，表明Flex已为端到端真实模型训练和推理做好准备。
英文摘要：
Sparsely-gated mixture-of-experts (MoE) has been widely adopted to scale deep learning models to trillion-plus parameters with fixed computational cost. The algorithmic performance of MoE relies on its token routing mechanism that forwards each input token to the right sub-models or experts. While token routing dynamically determines the amount of expert workload at runtime, existing systems suffer inefficient computation due to their static execution, namely static parallelism and pipelining, which does not adapt to the dynamic workload. We present Flex, a highly scalable stack design and implementation for MoE with dynamically adaptive parallelism and pipelining. Flex designs an identical layout for distributing MoE model parameters and input data, which can be leveraged by all possible parallelism or pipelining methods without any mathematical inequivalence or tensor migration overhead. This enables adaptive parallelism/pipelining optimization at zero cost during runtime. Based on this key design, Flex also implements various MoE acceleration techniques. Aggregating all techniques, Flex finally delivers huge speedup at any scale -- 4.96x and 5.75x speedup of a single MoE layer over 16 and 2,048 A100 GPUs, respectively, over the previous state-of-the-art. Our evaluation shows that Flex efficiently and effectively runs a real-world MoE-based model named SwinV2-MoE, built upon Swin Transformer V2, a state-of-the-art computer vision architecture. On efficiency, Flex accelerates SwinV2-MoE, achieving up to 1.55x and 2.11x speedup in training and inference over Fairseq, respectively. On effectiveness, the SwinV2-MoE model achieves superior accuracy in both pre-training and down-stream computer vision tasks such as COCO object detection than the counterpart dense model, indicating the readiness of Flex for end-to-end real-world model training and inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
