MoESys：面向互联网服务的分布式高效混合专家训练与推理系统
MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services
AI总结：
本文提出分布式 MoE 训练与推理系统 MoESys，通过 Elastic MoE、二维预取、分层融合通信及 CPU-GPU 环形内存轮询执行提升效率，并在 48 张 A100 上高效训练 12B 参数 UFO 模型。
摘要：
尽管聊天机器人、搜索引擎和在线广告等现代互联网服务需要使用大规模深度神经网络（DNN），但仍需要在异构计算系统上进行分布式训练和推理，以支撑这些 DNN 模型。混合专家（Mixture-of-Experts，MoE）是通过门控和并行化、以分而治之的方式，在模型/数据总体规模受限情况下降低训练成本的最常见策略之一。尽管 DeepSpeed 已致力于在异构基础设施上开展大规模 MoE 训练，但训练和推理效率仍可从负载均衡、通信/计算效率以及内存占用限制等多个系统层面进一步提升。在本工作中，我们提出了一种新颖的 MoESys，可同时提升大规模训练和推理的效率。具体而言，在训练过程中，所提出的 MoESys 采用 Elastic MoE 训练策略，结合二维预取和基于分层存储的 Fusion 通信，从而实现高效并行。为了在单节点内进行可扩展推理，尤其是当模型规模大于 GPU 内存时，MoESys 将 CPU-GPU 内存共同构建为一个由多个区段组成的环以加载模型，并以轮询方式跨内存区段执行计算任务，从而实现高效推理。我们开展了大量实验评估 MoESys，其中 MoESys 在 48 张 A100 GPU 卡上用 8 天成功训练了一个 Unified Feature Optimization（UFO）模型，该模型包含一个具有 12B 参数的稀疏门控混合专家模型。与最先进方法的比较表明，MoESys 总体上优于 DeepSpeed：训练吞吐量（每秒 token 数）提高 33%，推理吞吐量提高 13%。特别是在不均衡 MoE 任务（例如 UFO）下，MoESys 实现了 64% 的吞吐量提升，同时内存占用降低 18%。
英文摘要：
While modern internet services, such as chatbots, search engines, and online advertising, demand the use of large-scale deep neural networks (DNNs), distributed training and inference over heterogeneous computing systems are desired to facilitate these DNN models. Mixture-of-Experts (MoE) is one the most common strategies to lower the cost of training subject to the overall size of models/data through gating and parallelism in a divide-and-conquer fashion. While DeepSpeed has made efforts in carrying out large-scale MoE training over heterogeneous infrastructures, the efficiency of training and inference could be further improved from several system aspects, including load balancing, communication/computation efficiency, and memory footprint limits. In this work, we present a novel MoESys that boosts efficiency in both large-scale training and inference. Specifically, in the training procedure, the proposed MoESys adopts an Elastic MoE training strategy with 2D prefetch and Fusion communication over Hierarchical storage, so as to enjoy efficient parallelisms. For scalable inference in a single node, especially when the model size is larger than GPU memory, MoESys builds the CPU-GPU memory jointly into a ring of sections to load the model, and executes the computation tasks across the memory sections in a round-robin manner for efficient inference. We carried out extensive experiments to evaluate MoESys, where MoESys successfully trains a Unified Feature Optimization (UFO) model with a Sparsely-Gated Mixture-of-Experts model of 12B parameters in 8 days on 48 A100 GPU cards. The comparison against the state-of-the-art shows that MoESys outperformed DeepSpeed with 33% higher throughput (tokens per second) in training and 13% higher throughput in inference in general. Particularly, under unbalanced MoE Tasks, e.g., UFO, MoESys achieved 64% higher throughput with 18% lower memory footprints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
