Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token. Leading methods fall into two categories: analytic methods produce a converted model quickly using heuristics, while learning methods produce a model much more slowly by optimizing via gradients. Both approaches, however, fix the same active expert count, or "utilization," at every layer. We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers. Its name draws a metaphor between separating a dense FFN into experts and separating a light beam into its spectrum. Spectral-MoE consists of two distinct approaches: a static approach that uses a single per-layer scalar which is shared across tokens, and a dynamic approach which emits a per-token schedule. We demonstrate that both allocators improve performance over uniform on both the analytic-based and learning-based SOTA dense-to-MoE methods. At 25% FFN utilization, dynamic allocation reduces WikiText-2 perplexity relative to uniform by 1.04, 2.96, and 2.71 points on the learned base for Llama-2-7B, Llama-3-8B, and Qwen2.5-7B, respectively. The corresponding reductions on the analytic base are 2.45, 6.26, and 2.17 points. On the learned base, dynamic allocation improves six-task mean accuracy over uniform by an average of 1.5, 4.2, and 3.6 percentage points across the three models at FFN utilizations of 12.5%, 25%, and 50%, respectively. These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion.
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)
Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density
Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ, USA(计算机科学系,史蒂文斯理工学院,新泽西州霍博肯);Northeastern University, Boston, MA, USA(东北大学,波士顿,马萨诸塞州)
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China(计算机科学与技术学院,中国科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China, China(苏州先进研究院,中国科学技术大学)
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏滁研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Tongji University(同济大学);Cornell University(康奈尔大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));AI Training Platform Team, Shenzhen Loop Area Institute(深圳环域研究院AI训练平台团队)
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学);Chinese University of Hong Kong(中国香港大学)
Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
National Tsing Hua University(国立清华大学);National Taiwan University(国立台湾大学);University of Toronto(多伦多大学);NVIDIA AI Technology Center (NVAITC)(NVIDIA AI技术中心);Samsung Research America(三星美国研究院)
GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts
Department of Artificial Intelligence, Lviv Polytechnic National University(人工智能系,利沃夫工业大学);Department of Electrical Engineering, Eindhoven University of Technology(电气工程系,埃因霍温理工大学);The Bartlett School of Sustainable Construction, University College London(可持续建设学院,伦敦大学学院)
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所);Tübingen AI Center(图宾根人工智能中心);Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Xiamen University(厦门大学多媒体可信感知与高效计算重点实验室);International Computer Science Institute(国际计算机科学研究所);Lawrence Berkeley National Laboratory(伯克利国家实验室);University of California, Berkeley(加州大学伯克利分校);Liquid AI