本文提出了一种适用于混合专家（Mixture of Experts, MoE）模型的新型动态专家选择框架，旨在根据输入难度调整激活的专家数量，从而提升计算效率与模型性能。传统MoE方法依赖固定的Top-K路由机制，无论输入复杂度如何，都激活预设数量的专家；而我们的方法则根据每个输入在专家选择上的置信度水平动态选择专家。这一机制能更高效地利用计算资源：对于需要高级推理的复杂任务，激活更多专家；对于较简单的任务，则激活更少专家。通过大量评估，我们的动态路由方法在各类基准测试中相比传统Top-2路由实现了显著提升，在激活参数不足90%的情况下，平均提升幅度达0.7%。进一步分析表明，我们的模型会为需要复杂推理能力的任务（如BBH）分配更多专家，证实了其能根据输入复杂度动态分配计算资源的能力。我们的研究还发现，Transformer模型不同层所需的专家数量存在差异，这为设计异构MoE框架提供了思路。代码与模型可在https://github.com/ZhenweiAn/Dynamic_MoE获取。
英文摘要：
In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adjusting the number of activated experts based on input difficulty. Unlike traditional MoE approaches that rely on fixed Top-K routing, which activates a predetermined number of experts regardless of the input's complexity, our method dynamically selects experts based on the confidence level in expert selection for each input. This allows for a more efficient utilization of computational resources, activating more experts for complex tasks requiring advanced reasoning and fewer for simpler tasks. Through extensive evaluations, our dynamic routing method demonstrates substantial improvements over conventional Top-2 routing across various benchmarks, achieving an average improvement of 0.7% with less than 90% activated parameters. Further analysis shows our model dispatches more experts to tasks requiring complex reasoning skills, like BBH, confirming its ability to dynamically allocate computational resources in alignment with the input's complexity. Our findings also highlight a variation in the number of experts needed across different layers of the transformer model, offering insights into the potential for designing heterogeneous MoE frameworks. The code and models are available at https://github.com/ZhenweiAn/Dynamic_MoE.
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Department of Mathematics, Harvard University（哈佛大学数学系）
Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density
Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ, USA（计算机科学系，史蒂文斯理工学院，新泽西州霍博肯）；Northeastern University, Boston, MA, USA（东北大学，波士顿，马萨诸塞州）
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；University of Toronto（多伦多大学）；Vector Institue（向量研究所）；MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室）；Microsoft Research（微软研究院）；Mila（蒙特利尔人工智能研究院）
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA（计算机科学与软件工程系，阿伯拉罕大学，阿伯丁，阿拉巴马州，美国）；Department of Information Management, National Central University, Taoyuan, Taiwan（信息管理系，国立中央大学，桃园，台湾）
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Tongji University（同济大学）；Cornell University（康奈尔大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；AI Training Platform Team, Shenzhen Loop Area Institute（深圳环域研究院AI训练平台团队）
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fudan University, Shanghai, China（复旦大学）；University of Bath, Bath, United Kingdom（巴斯大学）；Oxford Suzhou Centre for Advanced Research, Suzhou, China（牛津苏滁研究中心）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Department of Computer Science, University of Colorado Boulder, Colorado, USA（计算机科学系，科罗拉多大学博尔德分校）；Department of Electrical Engineering and Computer Science, University of Michigan（电气工程与计算机科学系，密歇根大学）
When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry
Institute of Science Tokyo（东京科学大学）；Nanyang Technological University（南洋理工大学）；Tohoku University（东北大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Shandong University（山东大学）
