Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density
Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ, USA（计算机科学系，史蒂文斯理工学院，新泽西州霍博肯）；Northeastern University, Boston, MA, USA（东北大学，波士顿，马萨诸塞州）
Mixture-of-Experts (MoE) based Large Language Models (LLMs) have achieved superior performance, yet the massive memory overhead caused by storing multiple expert networks severely hinders their practical deployment. Singular Value Decomposition (SVD)-based compression has emerged as a promising post-training technique; however, most existing methods apply uniform rank allocation or rely solely on static weight properties. This overlooks the substantial heterogeneity in expert utilization observed in MoE models, where frequent routing patterns and intrinsic information density vary significantly across experts. In this work, we propose RFID-MoE, an effective framework for MoE compression by exploiting heterogeneous Routing Frequency and Information Density. We first introduce a fused metric that combines expert activation frequency with effective rank to measure expert importance, adaptively allocating higher ranks to critical expert groups under a fixed budget. Moreover, instead of discarding compression residuals, we reconstruct them via a parameter-efficient sparse projection mechanism to recover lost information with minimal parameter overhead. Extensive experiments on representative MoE LLMs (e.g., Qwen3, DeepSeekMoE) across multiple compression ratios demonstrate that RFID-MoE consistently outperforms state-of-the-art methods like MoBE and D2-MoE. Notably, RFID-MoE achieves a perplexity of 16.92 on PTB with the Qwen3-30B model at a 60% compression ratio, reducing perplexity by over 8.0 compared to baselines, and improves zero-shot accuracy on HellaSwag by approximately 8%.
Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Department of Mathematics, Harvard University（哈佛大学数学系）
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fudan University, Shanghai, China（复旦大学）；University of Bath, Bath, United Kingdom（巴斯大学）；Oxford Suzhou Centre for Advanced Research, Suzhou, China（牛津苏滁研究中心）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Department of Computer Science, University of Colorado Boulder, Colorado, USA（计算机科学系，科罗拉多大学博尔德分校）；Department of Electrical Engineering and Computer Science, University of Michigan（电气工程与计算机科学系，密歇根大学）
FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Tongji University（同济大学）；Cornell University（康奈尔大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；AI Training Platform Team, Shenzhen Loop Area Institute（深圳环域研究院AI训练平台团队）
When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry
Institute of Science Tokyo（东京科学大学）；Nanyang Technological University（南洋理工大学）；Tohoku University（东北大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Shandong University（山东大学）
When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
Department of Computer Science and Software Engineering（计算机科学与软件工程系）；Auburn University（阿伯拉罕大学）；Department of Information Management（信息管理系）；National Central University（国立中央大学）
Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
Nanjing University of Science and Technology（南京理工大学）；C 2 \text{C}^{2} DL, Institute of Automation, Chinese Academy of Sciences（C2 DL，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Computer Science, University of Sydney（悉尼大学计算机学院）
