OrchMLLM: 通过批量后平衡 orchestrate 多模态数据以加速多模态大语言模型训练
OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training
AI总结:
OrchMLLM通过批量后平衡调度器和全局协调器解决多模态数据训练中的模态不一致问题,提升训练效率和可扩展性。
摘要:
多模态大语言模型(MLLMs),如GPT-4o,正受到广泛关注。在探索MLLM训练过程中,我们识别出一种现象,即某种模态在不同示例中的比例变化显著。这加剧了处理小批量不平衡的挑战,导致Data Parallel(DP)实例之间的GPU利用率不均,并严重降低MLLM训练的效率和可扩展性,最终影响训练速度并阻碍进一步的MLLM研究。\n为了解决这些挑战,我们引入了OrchMLLM,一种综合框架,旨在减轻由模态组成不一致引起的MLLM训练低效问题。首先,我们提出了批量后平衡调度器,一种能够高效消除顺序数据中小型批次不平衡的技术。此外,我们将MLLM全局协调器整合到训练框架中,以协调多模态数据并解决由模态组成不一致带来的问题。我们评估了OrchMLLM在各种MLLM规模上的表现,证明了其效率和可扩展性。实验结果表明,当使用2560块H100 GPU训练84B MLLM时,OrchMLLM实现了41.6%的模型FLOPs利用率,比Megatron-LM在吞吐量上高出多达3.1倍。
英文摘要:
Multimodal large language models (MLLMs), such as GPT-4o, are garnering significant attention. During the exploration of MLLM training, we identified Modality Composition Incoherence, a phenomenon that the proportion of a certain modality varies dramatically across different examples. It exacerbates the challenges of addressing mini-batch imbalances, which lead to uneven GPU utilization between Data Parallel (DP) instances and severely degrades the efficiency and scalability of MLLM training, ultimately affecting training speed and hindering further research on MLLMs. To address these challenges, we introduce OrchMLLM, a comprehensive framework designed to mitigate the inefficiencies in MLLM training caused by Modality Composition Incoherence. First, we propose Batch Post-Balancing Dispatcher, a technique that efficiently eliminates mini-batch imbalances in sequential data. Additionally, we integrate MLLM Global Orchestrator into the training framework to orchestrate multimodal data and tackle the issues arising from Modality Composition Incoherence. We evaluate OrchMLLM across various MLLM sizes, demonstrating its efficiency and scalability. Experimental results reveal that OrchMLLM achieves a Model FLOPs Utilization (MFU) of $41.6\%$ when training an 84B MLLM with three modalities on $2560$ H100 GPUs, outperforming Megatron-LM by up to $3.1\times$ in throughput.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。