arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.07067cs.LGcs.DC
收藏

MoE-CAP:稀疏专家混合系统成本、准确率和性能的基准测试

MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems

University of Edinburgh(爱丁堡大学);Microsoft Research(微软研究院);Peking University(北京大学);NVIDIA Co-leading authors(NVIDIA)

Yinsicheng Jiang, Yao Fu, Yeqi Huang, Ping Nie, Zhan Lu, Leyang Xue, Congjie He, Man-Kit Sit, Jilong Xue, Li Dong, Ziming Miao, Dayou Du, Tairan Xu, Kai Zou, Edoardo Ponti, Luo Mai

AI总结:

MoE-CAP通过引入CAP雷达图和稀疏性能指标,系统性地评估稀疏混合专家系统在成本、准确率和性能之间的权衡。

摘要:

稀疏混合专家(MoE)架构正日益被用于高效扩展大型语言模型(LLMs),但其依赖于异构的计算和内存资源。这些因素共同影响系统成本、准确率和性能(CAP),使权衡成为必然。现有基准测试往往无法准确捕捉这些权衡,使实际部署决策变得复杂。为此,我们引入了MoE-CAP,专门针对MoE系统设计的基准测试。我们的分析表明,当前硬件下难以在CAP之间取得最佳平衡;MoE系统通常在优化两个维度时牺牲第三个维度,这种动态我们称为MoE-CAP权衡。为了可视化这一点,我们提出了CAP雷达图。我们进一步引入了意识稀疏的性能指标——稀疏内存带宽利用率(S-MBU)和稀疏模型FLOPS利用率(S-MFU)——以在不同硬件平台和部署场景下准确评估MoE系统的性能。

英文摘要:

The sparse Mixture-of-Experts (MoE) architecture is increasingly favored for scaling Large Language Models (LLMs) efficiently, but it depends on heterogeneous compute and memory resources. These factors jointly affect system Cost, Accuracy, and Performance (CAP), making trade-offs inevitable. Existing benchmarks often fail to capture these trade-offs accurately, complicating practical deployment decisions. To address this, we introduce MoE-CAP, a benchmark specifically designed for MoE systems. Our analysis reveals that achieving an optimal balance across CAP is difficult with current hardware; MoE systems typically optimize two of the three dimensions at the expense of the third-a dynamic we term the MoE-CAP trade-off. To visualize this, we propose the CAP Radar Diagram. We further introduce sparsity-aware performance metrics-Sparse Memory Bandwidth Utilization (S-MBU) and Sparse Model FLOPS Utilization (S-MFU)-to enable accurate performance benchmarking of MoE systems across diverse hardware platforms and deployment scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑