密集训练、稀疏推理：重新思考混合专家语言模型的训练
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
AI总结：
本文提出DS-MoE框架，训练时密集使用全部专家、推理时稀疏激活，以30-40%参数激活实现与密集模型相当的性能，并显著提升生成速度。
摘要：
混合专家（MoE）语言模型能够在不牺牲性能的情况下，将计算成本相比密集模型降低2-4倍，因此在计算受限场景中更高效。然而，MoE模型通常需要2-4倍的参数才能达到与密集模型相当的性能，这会带来更大的GPU内存需求，并使MoE模型在自回归生成等I/O受限场景中效率较低。在本工作中，我们提出了一种用于MoE模型的混合密集训练与稀疏推理框架（DS-MoE），该框架通过在训练期间对所有专家采用密集计算、在推理期间采用稀疏计算，实现了出色的计算效率和参数效率。我们在训练LLM方面的实验表明，DS-MoE模型比标准稀疏MoE更具参数效率，并且在总参数规模和性能上与密集模型相当，同时计算成本更低（仅激活模型30-40%的参数）。使用vLLM进行的性能测试显示，我们的DS-MoE-6B模型运行速度最高可达Mistral-7B等类似密集模型的1.86倍，并且比DeepSeekMoE-16B和Qwen1.5-MoE-A2.7B等可比MoE快1.50倍至1.71倍。
英文摘要：
Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4$\times$ compared to dense models without sacrificing performance, making them more efficient in computation-bounded scenarios. However, MoE models generally require 2-4$\times$ times more parameters to achieve comparable performance to a dense model, which incurs larger GPU memory requirements and makes MoE models less efficient in I/O-bounded scenarios like autoregressive generation. In this work, we propose a hybrid dense training and sparse inference framework for MoE models (DS-MoE) which achieves strong computation and parameter efficiency by employing dense computation across all experts during training and sparse computation during inference. Our experiments on training LLMs demonstrate that our DS-MoE models are more parameter-efficient than standard sparse MoEs and are on par with dense models in terms of total parameter size and performance while being computationally cheaper (activating 30-40% of the model's parameters). Performance tests using vLLM show that our DS-MoE-6B model runs up to $1.86\times$ faster than similar dense models like Mistral-7B, and between $1.50\times$ and $1.71\times$ faster than comparable MoEs, such as DeepSeekMoE-16B and Qwen1.5-MoE-A2.7B.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
