MoE-DisCo：低经济成本训练混合专家模型
MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
AI总结：
MoE-DisCo通过分阶段训练框架，利用低成本硬件降低混合专家模型训练成本，同时在多个任务上实现性能提升。
摘要：
训练大规模混合专家（MoE）模型通常需要高内存、高带宽的GPU（例如A100），其高昂的成本已成为大模型训练的主要障碍。相比之下，经济型硬件成本低，但受限于内存容量和带宽，无法直接用于大语言模型训练。为此，我们提出了MoE-DisCo（混合专家与解耦聚类与协调），一种分阶段训练框架。MoE-DisCo将MoE模型分解为多个密集子模型，每个子模型由共享主干和单个专家组成，并使用无监督聚类将训练数据分成子集。每个子模型独立且并行地在其分配的数据子集上使用低成本设备进行训练，无需任何设备间通信。随后，所有专家被整合到完整的MoE模型中，并在高内存、高带宽GPU上进行短期全局微调。实验表明，我们的方法在多个下游任务、损失函数和困惑度（PPL）上与全参数训练匹配或甚至超越，同时在Qwen1.5-MoE-2.7B和Llama-MoE-3.5B等不同数据集上将训练成本降低了47.6%至69.5%。
英文摘要：
Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model training. In contrast, affordable hardware is low-cost but constrained by memory capacity and bandwidth, making it unsuitable for direct LLM training. To address this, we propose MoE-DisCo (Mixture-of-Experts with Disentangled Clustering and Coordination), a staged training framework. MoE-DisCo decomposes the MoE model into multiple dense submodels, each consisting of a shared backbone and a single expert, and partitions the training data into subsets using unsupervised clustering. Each submodel is trained independently and in parallel on its assigned data subset using low-cost devices, without any inter-device communication. Subsequently, all experts are integrated into a complete MoE model and fine-tuned globally for a short period on high-memory, high-bandwidth GPUs. Experiments show that our method matches or even surpasses full-parameter training in performance across multiple downstream tasks, loss function, and perplexity (PPL), while reducing training cost by 47.6 percent to 69.5 percent on Qwen1.5-MoE-2.7B and Llama-MoE-3.5B across different datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
