Optimus：通过气泡利用加速大规模多模态大语言模型训练
Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
AI总结：
本文提出Optimus分布式训练系统，通过在LLM气泡内调度编码器计算并优化亚毫秒级气泡调度，在3072个GPU上将MLLM训练加速20.5%-21.3%。
摘要：
多模态大语言模型（MLLMs）将大语言模型（LLMs）的成功扩展到图像、文本和音频等多种数据类型，在多模态翻译、视觉问答和内容生成等多个领域取得了显著性能。然而，由于异构模态模型和三维并行中复杂的数据依赖关系导致大量GPU气泡，现有系统训练MLLMs的效率较低。本文提出Optimus，一个分布式MLLM训练系统，旨在缩短端到端MLLM训练时间。Optimus基于我们的原则性分析，即在LLM气泡内调度编码器计算可以减少MLLM训练中的气泡。为了使所有GPU都能调度编码器计算，Optimus为编码器和LLM分别搜索独立的并行计划，并采用气泡调度算法，在不破坏MLLM模型架构中原始数据依赖关系的前提下利用LLM气泡。我们进一步将编码器层计算分解为一系列内核，并分析三维并行的常见气泡模式，以精心优化亚毫秒级气泡调度，从而最小化整体训练时间。我们在生产集群中的实验表明，与基线相比，Optimus在3072个GPU上使用ViT-22B和GPT-175B模型将MLLM训练加速了20.5%-21.3%。
英文摘要：
Multimodal large language models (MLLMs) have extended the success of large language models (LLMs) to multiple data types, such as image, text and audio, achieving significant performance in various domains, including multimodal translation, visual question answering and content generation. Nonetheless, existing systems are inefficient to train MLLMs due to substantial GPU bubbles caused by the heterogeneous modality models and complex data dependencies in 3D parallelism. This paper proposes Optimus, a distributed MLLM training system that reduces end-to-end MLLM training time. Optimus is based on our principled analysis that scheduling the encoder computation within the LLM bubbles can reduce bubbles in MLLM training. To make scheduling encoder computation possible for all GPUs, Optimus searches the separate parallel plans for encoder and LLM, and adopts a bubble scheduling algorithm to enable exploiting LLM bubbles without breaking the original data dependencies in the MLLM model architecture. We further decompose encoder layer computation into a series of kernels, and analyze the common bubble pattern of 3D parallelism to carefully optimize the sub-millisecond bubble scheduling, minimizing the overall training time. Our experiments in a production cluster show that Optimus accelerates MLLM training by 20.5%-21.3% with ViT-22B and GPT-175B model over 3072 GPUs compared to baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
