Alloc-MoE：基于预算的专家激活分配以实现高效的混合专家推理
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
AI总结：
本文提出Alloc-MoE框架，通过层和token层面协调优化激活预算分配，减少推理延迟并保持性能，实验显示在DeepSeek-V2-Lite上实现1.15倍的预填和1.34倍的解码加速。
摘要：
混合专家（MoE）因其稀疏激活机制已成为扩展大语言模型的主要架构。然而，大量专家激活导致推理过程中的显著延迟瓶颈，尤其是在资源受限的部署场景中。现有减少专家激活的方法可能导致模型性能严重退化。本文引入了激活预算的概念作为专家激活数量的约束，并提出Alloc-MoE框架，通过层和token层面协调优化预算分配，以最小化性能退化。在层层面，我们引入了Alloc-L，利用敏感性分析和动态规划确定各层专家激活的最优分配。在token层面，我们提出Alloc-T，根据路由分数动态重新分配激活，优化预算分配而不增加延迟。在多个MoE模型上的广泛实验表明，Alloc-MoE在受限的激活预算下保持模型性能。特别是，在DeepSeek-V2-Lite上，Alloc-MoE在原始预算的一半情况下实现了1.15倍的预填加速和1.34倍的解码加速。
英文摘要：
Mixture-of-Experts (MoE) has become a dominant architecture for scaling large language models due to their sparse activation mechanism. However, the substantial number of expert activations creates a critical latency bottleneck during inference, especially in resource-constrained deployment scenarios. Existing approaches that reduce expert activations potentially lead to severe model performance degradation. In this work, we introduce the concept of \emph{activation budget} as a constraint on the number of expert activations and propose Alloc-MoE, a unified framework that optimizes budget allocation coordinately at both the layer and token levels to minimize performance degradation. At the layer level, we introduce Alloc-L, which leverages sensitivity profiling and dynamic programming to determine the optimal allocation of expert activations across layers. At the token level, we propose Alloc-T, which dynamically redistributes activations based on routing scores, optimizing budget allocation without increasing latency. Extensive experiments across multiple MoE models demonstrate that Alloc-MoE maintains model performance under a constrained activation budget. Especially, Alloc-MoE achieves $1.15\times$ prefill and $1.34\times$ decode speedups on DeepSeek-V2-Lite at half of the original budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
