MegaScale-Omni: 一种面向多模态大语言模型生产训练的超大规模、工作负载容错系统
MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production
AI总结:
本文提出MegaScale-Omni系统,通过解耦并行策略、统一表示和负载平衡技术,提升多模态大语言模型在动态工作负载下的训练效率,实现1.27倍至7.57倍的吞吐量提升。
摘要:
作为多功能AI应用的基础组件,训练多模态大语言模型(MLLM)依赖于具有动态模态混合比例和样本长度分布的多模态数据集。然而,现有MLLM系统在动态工作负载下效率低下,因为编码器与LLM主干之间的资源分配和模型并行化决策是静态耦合的。本文提出了MegaScale-Omni,一种面向动态工作负载适应和超大规模部署的工业级MLLM训练系统。MegaScale-Omni基于编码器-LLM复用训练方案,有三个关键创新:(1)解耦并行策略,利用长-短序列并行处理可变长度样本,以及针对LLM主干的完整5D并行策略,均组织在高效的通信并行布局下。(2)统一的编码器-LLM表示,实现灵活可扩展的协同定位,以及新的编码器-LLM联合流水线模式,具有工作负载容错能力。(3)通过去中心化的分组重排序数据加载器和从编码器到LLM排名的自适应重分区技术实现负载平衡。MegaScale-Omni被部署为我们在内部大规模MLLM训练任务的基础,实验结果表明,与四个最先进的系统相比,在生产级动态工作负载下,其吞吐量提升了1.27倍至7.57倍。
英文摘要:
As the foundational component of versatile AI applications, training an multimodal large language model (MLLM) relies on multimodal datasets with dynamic modality mixture proportions and sample length distributions. However, existing MLLM systems remain inefficient under dynamic workloads, due to statically coupled decisions of resource allocation and model parallelization between encoders and the LLM backbone. This paper presents MegaScale-Omni, an industrial-grade MLLM training system tailored for dynamic workload adaption and hyper-scale deployment. MegaScale-Omni is built upon the training scheme of encoder-LLM multiplexing with three key innovations: (1) Decoupled parallelism strategies with long-short sequence parallelism for encoders to process variable-length samples, and full-fledged 5D parallelism for the LLM backbone, both organized under a communication-efficient parallelization layout. (2) Unified encoder-LLM representations for flexible, extensible colocation, and a new paradigm of encoder-LLM joint pipeline with workload resilience. (3) Workload balancing techniques via decentralized grouped reordering in data loaders and adaptive resharding from encoder to LLM ranks. MegaScale-Omni is deployed as the foundation of our in-house large-scale MLLM training tasks with thousands of GPUs. Our experimental results demonstrate $1.27\times$-$7.57\times$ throughput improvement under production-grade dynamic workloads, as compared to four state-of-the-art systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。