Sparse Mixture-of-Experts (SMoE) language models achieve strong capability at low per-token compute, yet deployment remains constrained by memory footprint and throughput because the full expert pool must still be stored and served. Post-training expert pruning reduces this cost, but most methods focus on which experts to prune within each layer and default to a uniform layer-wise sparsity allocation, even though the layer-wise allocation can strongly affect performance. We decouple pruning into within-layer expert ranking and across-layer budget allocation, and introduce \textbf{E}xpected \textbf{S}peculative \textbf{A}cceptance \textbf{P}roxy (\textbf{ESAP}), a speculative-decoding-inspired, teacher-forced metric that measures how well a pruned model matches the full model without costly autoregressive decoding. ESAP is bounded and stable, enabling cheap comparison of many candidates. Building on ESAP, we propose EvoESAP, an evolutionary search framework that finds an improved non-uniform layer-wise sparsity allocation under a fixed global budget while holding the within-layer pruning order fixed, making it a plug-and-play method for criteria such as Frequency, EAN, SEER, and REAP. Across 7B--30B SMoE LLMs at 25\% and 50\% sparsity, EvoESAP consistently discovers non-uniform allocations that improve open-ended generation (up to \textbf{+19.6\%} on MATH-500 at 50\% sparsity) while preserving competitive multiple-choice accuracy compared with uniform pruning at the same sparsity. Code is available at https://github.com/ZongfangLiu/EvoESAP.
How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
Zhejiang University（浙江大学）；Westlake University（西湖大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Carnegie Mellon University（卡内基梅隆大学）
Zhejiang University（浙江大学）；Westlake University（西湖大学）；Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China（计算机科学与技术学院，中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China, China（苏州先进研究院，中国科学技术大学）
Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Nanyang Technological University（南洋理工大学）
Beihang University（北京航空航天大学）；The Institute of Acoustics of the Chinese Academy of Sciences（中国科学院声学研究所）；Microsoft（微软）
The sparse Mixture-of-Experts (MoE) model is powerful for large-scale pre-training and has achieved promising results due to its model capacity. However, with trillions of parameters, MoE is hard to be deployed on cloud or mobile environment. The inference of MoE requires expert parallelism, which is not hardware-friendly and communication expensive. Especially for resource-limited downstream tasks, such sparse structure has to sacrifice a lot of computing efficiency for limited performance gains. In this work, we observe most experts contribute scarcely little to the MoE fine-tuning and infer…
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）；University of Science and Technology of China（中国科学技术大学）
Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
University of Surrey（萨里大学）；Clemson University（克莱姆森大学）；Meituan（美团）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sun Yat-sen University（中山大学）；The University of Arizona（亚利桑那大学）；ELLIS Institute（ELLIS研究所）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；Tübingen AI Center（图宾根人工智能中心）
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Nanjing University of Science and Technology（南京理工大学）；AIRIA
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；Fudan University（复旦大学）；Chinese University of Hong Kong（中国香港大学）
Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；University of International Business and Economics（国际商务经济大学）；YanTron Technology Co. Ltd（燕tron技术有限公司）；EBTech Co. Ltd（EBTech有限公司）
Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea（电气与计算机工程系，首尔国立大学，首尔，韩国）；Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea（人工智能跨学科项目，首尔国立大学，首尔，韩国）
Mila – Quebec AI Institute（Mila – 魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔综合理工学院）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；AI Center, Samsung, South Korea（三星人工智能中心，韩国）；Samsung AI Lab, Montreal（三星人工智能实验室，蒙特利尔）
The Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast with dense models, the sparse architecture of MoE offers opportunities for drastically growing model size with significant accuracy gain while consuming much lower compute budget. However, supporting large scale MoE training also has its own set of system and modeling challenges. To overcome the challenges and embrace the opportunities of MoE, we first develop a system capable of scaling MoE models efficiently to tr…
