TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration
Institute for Artificial Intelligence（人工智能研究院）；School of Integrated Circuits（集成电路学院）；Yuanpei College（元培学院）；Beijing Advanced Innovation Center for Integrated Circuits（北京集成电路先进创新中心）
Diffusion large language models (dLLMs) have recently gained significant attention due to their inherent support for parallel decoding. Building on this paradigm, Mixture-of-Experts (MoE) dLLMs with autoregressive (AR) initialization have further demonstrated strong performance competitive with mainstream AR models. However, we identify a fundamental mismatch between MoE architectures and diffusion-based decoding. Specifically, a large number of experts are activated at each denoising step, while only a small subset of tokens is ultimately accepted, resulting in substantial inference overhead and limiting their deployment in latency-sensitive applications. In this work, we propose TEAM, a plug-and-play framework that accelerates MoE dLLMs by enabling more accepted tokens with fewer activated experts. TEAM is motivated by the observation that expert routing decisions exhibit strong temporal consistency across denoising levels as well as spatial consistency across token positions. Leveraging these properties, TEAM employs three complementary expert activation and decoding strategies, conservatively selecting necessary experts for decoded and masked tokens and simultaneously performing aggressive speculative exploration across multiple candidates. Experimental results demonstrate that TEAM achieves up to 2.2x speedup over vanilla MoE dLLM, with negligible performance degradation. Code is released at https://github.com/PKU-SEC-Lab/TEAM-MoE-dLLM.
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；Fudan University（复旦大学）；Chinese University of Hong Kong（中国香港大学）
What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models
Guangdong University of Technology（广东工业大学）；Soochow University（苏州大学）；University of Copenhagen（哥本哈根大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology（国防科技大学并行与分布计算国家重点实验室，计算机科学与技术学院）
Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
The Chinese University of Hong Kong（香港中文大学）；Renmin University of China（中国人民大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Westlake University（西湖大学）
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
Soochow University（苏州大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；KTH Royal Institute of Technology（瑞典皇家理工学院）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Nanjing College, University of Chinese Academy of Sciences（中国科学院大学南京学院）；Nanjing Institute of Information Superbahn（南京信息超级bahn研究院）；Dobot Robotics（越疆机器人）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉科学学院）；University of Chinese Academy of Sciences（中国科学院大学）
Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
Center for Information and Language Processing (CIS)（信息与语言处理中心）；Ubiquitous Knowledge Processing Lab (UKP)（无所不在的知识处理实验室）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts
Tencent AI Lab（腾讯人工智能实验室）
近年来，基于Transformer的混合专家算法（Mixture of Experts，MoE）在许多领域都取得了可喜的成果。这在很大程度上是由于该架构的以下优点：首先，基于MoE的Transformer可以增加模型容量，而不会增加训练和推理时的计算成本。此外，基于MoE的Transformer是一种动态网络，可以适应实际应用中输入实例复杂性的变化。在这项工作中，我们探讨了基于MoE的语音识别模型，命名为Speech MoE。为了进一步控制路由器激活的稀疏性，提高门值的多样性，我们分别提出了稀疏性L1损失和平均重要性损失。此外，一个新的路由器架构中使用的语音MoE，它可以同时利用信息从一个共享的嵌入式网络和不同的MoE层的分层表示。实验结果表明，与传统静态网络相比，SpeechMoE在计算成本相当的情况下，可以获得更低的字符出错率（CER），在四个评估数据集上的CER相对改善为7.0%-23.0%。
Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Baidu Inc.（百度公司）
