Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world deployment. The Mixture of Experts (MoE) architecture offers a promising solution by dynamically selecting and activating only a subset of experts during inference, thus substantially reducing computational costs while preserving high performance. Despite these benefits, MoE introduces new inefficiencies, such as excessive parameters and communication overhead. In this work, we present a holistic study of compression techniques for Mixture of Experts to enhance both efficiency and scalability. While recent efforts have focused on Expert Trimming, which reduces the number of experts, these approaches still suffer from considerable communication and computational costs. To address this, we propose more aggressive strategies, such as Layer Drop, which removes entire MoE layers, and Block Drop, which eliminates transformer blocks. Surprisingly, these aggressive pruning techniques not only preserve model performance but also substantially improve computation and memory efficiency. Furthermore, beyond Expert Trimming, we also introduce Expert Slimming, which compresses individual experts to further boost performance and can be seamlessly integrated with Expert Trimming. Extensive experimental results demonstrate the effectiveness of our proposed methods-Layer Drop and Block Drop-along with the comprehensive recipe that integrates Expert Slimming and Expert Trimming, achieving a 6.05x speedup with 77.1% reduced memory usage while maintaining over 92% of performance on Mixtral-8x7B. Our code is released at https://github.com/CASE-Lab-UMD/Unified-MoE-Compression.
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Hi Lab, Xiaohongshu Inc.(小红书公司Hi实验室);Project Numina(项目Numina);School of Data Science, Fudan University(复旦大学数据科学学院)
MC#: Mixture Compressor for Mixture-of-Experts Large Models
Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电子与电气工程系);School of Computing, National University of Singapore(新加坡国立大学计算机学院);NVIDIA Research(NVIDIA研究)
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA(计算机科学与软件工程系,阿伯拉罕大学,阿伯丁,阿拉巴马州,美国);Department of Information Management, National Central University, Taoyuan, Taiwan(信息管理系,国立中央大学,桃园,台湾)
Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea(电气与计算机工程系,首尔国立大学,首尔,韩国);Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea(人工智能跨学科项目,首尔国立大学,首尔,韩国)
Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach
Division of Electrical and Computer Engineering, Louisiana State University(电气与计算机工程系,路易斯安那州立大学);Division of Computer Science and Engineering, Louisiana State University(计算机科学与工程系,路易斯安那州立大学);Department of Data Science, New Jersey Institute of Technology(数据科学系,新泽西理工学院);Department of Electrical and Computer Engineering, University of Louisiana at Lafayette(电气与计算机工程系,路易斯安那大学拉斐特分校);Department of Electrical and Electronic Engineering, Imperial College London(电气与电子工程系,伦敦帝国理工学院)
DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models
School of Information Science and Technology, Fudan University(复旦大学信息科学与技术学院);The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);Baidu USA(美国百度公司);StepFun(阶跃星辰)
MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
School of Information Sciences, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校信息科学学院);Hong Kong Baptist University(香港浸会大学);Jacobs Technion-Cornell Institute, Cornell Tech(康奈尔理工学院雅各布斯以色列理工学院)
Mixture Compressor for Mixture-of-Experts LLMs Gains More
The University of Hong Kong(香港大学);The Chinese University of Hong Kong(香港中文大学);Beihang University(北京航空航天大学);Centre for Perceptual and Interactive Intelligence(感知与交互智能中心)