动态专家共享：解耦内存与并行性在混合专家扩散语言模型中的方法
Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs
AI总结：
本文提出动态专家共享方法，通过序列层面的coreset选择，减少MoE dLLMs中专家激活数量和延迟，同时保持高准确性。
摘要：
在并行解码范式中，扩散大语言模型（dLLMs）已成为一种有前景的候选者，能够在生成质量和吞吐量之间取得平衡。然而，其与混合专家（MoE）架构的结合受到专家爆炸的限制：随着并行生成的token数量增加，激活的distinct专家数量几乎线性增长。这导致了显著的内存交通，将推理推入内存受限的领域，抵消了MoE和并行解码的效率提升。为了解决这一挑战，我们提出了动态专家共享（DES），一种新的技术，将MoE优化从以token为中心的剪枝和传统专家跳过方法转移到序列层面的coreset选择。为了最大化专家重用，DES识别一个紧凑、高效用的专家集，以满足整个并行解码块的要求。我们引入了两种创新的选取策略：（1）序列内共享（DES-Seq），它适应最优分配到序列层面；（2）显著性感知投票（DES-Vote），一种新的机制，允许token通过聚合路由权重共同选举一个coreset。在MoE dLLMs上的广泛实验表明，DES将唯一专家激活减少超过55%，并将延迟减少高达38%，同时保留99%的原始准确性，有效地解耦了内存开销与并行度。
英文摘要：
Among parallel decoding paradigms, diffusion large language models (dLLMs) have emerged as a promising candidate that balances generation quality and throughput. However, their integration with Mixture-of-Experts (MoE) architectures is constrained by an expert explosion: as the number of tokens generated in parallel increases, the number of distinct experts activated grows nearly linearly. This results in substantial memory traffic that pushes inference into a memory-bound regime, negating the efficiency gains of both MoE and parallel decoding. To address this challenge, we propose Dynamic Expert Sharing (DES), a novel technique that shifts MoE optimization from token-centric pruning and conventional expert skipping methods to sequence-level coreset selection. To maximize expert reuse, DES identifies a compact, high-utility set of experts to satisfy the requirements of an entire parallel decoding block. We introduce two innovative selection strategies: (1) Intra-Sequence Sharing (DES-Seq), which adapts optimal allocation to the sequence level, and (2) Saliency-Aware Voting (DES-Vote), a novel mechanism that allows tokens to collectively elect a coreset based on aggregated router weights. Extensive experiments on MoE dLLMs demonstrate that DES reduces unique expert activations by over 55% and latency by up to 38%, while retaining 99% of vanilla accuracy, effectively decoupling memory overhead from the degree of parallelism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
