FlexMoE：通过动态设备放置扩展大规模稀疏预训练模型训练
FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
AI总结：
针对稀疏MoE模型训练中的路由不平衡与波动问题，提出FlexMoE框架，通过动态专家管理和设备放置机制优化数据流，在真实负载下性能显著优于DeepSpeed和FasterMoE。
摘要：
随着数据量的不断增加，将知识存储于海量模型参数中的大规模预训练模型成为趋势。这些模型的训练由大量密集代数运算组成，需要庞大的硬件资源。近期，稀疏门控混合专家模型日益流行，并在多种下游任务中展现出令人印象深刻的预训练可扩展性。然而，由于路由不平衡和波动问题，这种稀疏条件计算在实际系统中的效果可能不及预期。通常，MoEs 正在成为数据生命周期中的一种新数据分析范式，并面临前所未有的规模、复杂性和粒度挑战。本文提出了一种新型 DNN 训练框架 FlexMoE，系统且透明地解决动态数据流引发的低效问题。我们首先对训练 MoE 模型的问题与机遇进行了实证分析，这促使我们通过动态专家管理和设备放置机制来克服路由不平衡与波动问题。随后，我们在现有 DNN 运行时上引入了新型调度模块，用于监控数据流、制定调度计划，并在实时数据流量的引导下动态调整模型到硬件的映射。我们利用简单高效的启发式算法，在训练过程中动态优化设备放置。我们在 NLP 模型（如 BERT 和 GPT）及视觉模型（如 Swin）上均进行了实验。结果表明，在真实工作负载下，FlexMoE 相比现有系统取得了卓越性能：平均优于 DeepSpeed 1.70 倍，最高达 2.10 倍；平均优于 FasterMoE 1.30 倍，最高达 1.45 倍。
英文摘要：
With the increasing data volume, there is a trend of using large-scale pre-trained models to store the knowledge into an enormous number of model parameters. The training of these models is composed of lots of dense algebras, requiring a huge amount of hardware resources. Recently, sparsely-gated Mixture-of-Experts (MoEs) are becoming more popular and have demonstrated impressive pretraining scalability in various downstream tasks. However, such a sparse conditional computation may not be effective as expected in practical systems due to the routing imbalance and fluctuation problems. Generally, MoEs are becoming a new data analytics paradigm in the data life cycle and suffering from unique challenges at scales, complexities, and granularities never before possible. In this paper, we propose a novel DNN training framework, FlexMoE, which systematically and transparently address the inefficiency caused by dynamic dataflow. We first present an empirical analysis on the problems and opportunities of training MoE models, which motivates us to overcome the routing imbalance and fluctuation problems by a dynamic expert management and device placement mechanism. Then we introduce a novel scheduling module over the existing DNN runtime to monitor the data flow, make the scheduling plans, and dynamically adjust the model-to-hardware mapping guided by the real-time data traffic. A simple but efficient heuristic algorithm is exploited to dynamically optimize the device placement during training. We have conducted experiments on both NLP models (e.g., BERT and GPT) and vision models (e.g., Swin). And results show FlexMoE can achieve superior performance compared with existing systems on real-world workloads -- FlexMoE outperforms DeepSpeed by 1.70x on average and up to 2.10x, and outperforms FasterMoE by 1.30x on average and up to 1.45x.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
