MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
University of Edinburgh(爱丁堡大学);Microsoft Research(微软研究院);Peking University(北京大学);NVIDIA Co-leading authors(NVIDIA)
Yinsicheng Jiang, Yao Fu, Yeqi Huang, Ping Nie, Zhan Lu, Leyang Xue, Congjie He, Man-Kit Sit, Jilong Xue, Li Dong, Ziming Miao, Dayou Du, Tairan Xu, Kai Zou, Edoardo Ponti, Luo Mai
The sparse Mixture-of-Experts (MoE) architecture is increasingly favored for scaling Large Language Models (LLMs) efficiently, but it depends on heterogeneous compute and memory resources. These factors jointly affect system Cost, Accuracy, and Performance (CAP), making trade-offs inevitable. Existing benchmarks often fail to capture these trade-offs accurately, complicating practical deployment decisions. To address this, we introduce MoE-CAP, a benchmark specifically designed for MoE systems. Our analysis reveals that achieving an optimal balance across CAP is difficult with current hardware; MoE systems typically optimize two of the three dimensions at the expense of the third-a dynamic we term the MoE-CAP trade-off. To visualize this, we propose the CAP Radar Diagram. We further introduce sparsity-aware performance metrics-Sparse Memory Bandwidth Utilization (S-MBU) and Sparse Model FLOPS Utilization (S-MFU)-to enable accurate performance benchmarking of MoE systems across diverse hardware platforms and deployment scenarios.
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China(计算机科学与技术学院,中国科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China, China(苏州先进研究院,中国科学技术大学)
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
School of Software, Tsinghua University, Beijing, China(清华大学软件学院,北京,中国);TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院,中国);School of New Media and Communication, Tianjin University, China(天津大学新媒体与传播学院,中国)
Zhejiang University(浙江大学);Westlake University(西湖大学);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers
School of Computer Science, Peking University(北京大学计算机科学学院);School of Software Engineering, University of Electronic Science and Technology of China(电子科技大学软件工程学院)
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Tongji University(同济大学);Cornell University(康奈尔大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));AI Training Platform Team, Shenzhen Loop Area Institute(深圳环域研究院AI训练平台团队)
College of Cyber Security, Jinan University(暨南大学网络空间安全学院);College of Mathematics and Computer Science, Shantou University(汕头大学数学与计算机科学学院);School of Information Engineering, Guangdong Eco-Engineering Polytechnic(广东生态工程职业学院信息工程学院);Department of Computer Science, University of Illinois Chicago(芝加哥伊利诺伊大学计算机科学系)
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学);Chinese University of Hong Kong(中国香港大学)
MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室)
Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea(电气与计算机工程系,首尔国立大学,首尔,韩国);Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea(人工智能跨学科项目,首尔国立大学,首尔,韩国)