qs不等式：混合专家模型在推理中的双重惩罚量化
The $qs$ Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inference
AI总结：
qs不等式量化了混合专家模型在推理中的双重惩罚，揭示其在长上下文场景下的性能劣势，并指出蒸馏到密集模型可能是提升推理效率的方法。
摘要：
混合专家（MoE）模型在低训练FLOPs下提供高质量，但这种效率在推理时往往消失。我们识别出一种双重惩罚，结构上使MoE架构在解码过程中处于不利地位：首先，专家路由会碎片化微批次并减少权重重用；其次，庞大的驻留专家池会减少高速缓存（HBM）的头寸，用于KV缓存。这种现象被正式定义为重用碎片化，迫使前馈网络（FFN）进入带宽受限的领域，特别是在长上下文长度时。我们引入了qs不等式，这是一个预测标准，用于识别MoE相对于质量匹配的密集模型在结构上处于劣势的情况。该标准统一了稀疏性（s），即每个token激活的参数比例，和质量等价因子（q），即密集模型需要的大小乘数以匹配MoE性能。我们在前沿模型中进行了评估，包括DeepSeek-V3、Qwen3-235B、Grok-1和Switch-C，证明这种碎片化是一种普遍的架构现象。对于DeepSeek-V3在128k上下文的情况下，这导致质量匹配的密集基线在吞吐量上具有4.5倍的优势。至关重要的是，大规模架构如Switch-C在集群规模中可能变得不可行，而质量匹配的密集模型仍然可行。我们的结果表明，训练时的FLOP效率是长上下文服务推理性能的不完整代理。它们还表明，MoE可能最好被视为训练时的优化，通过蒸馏到密集模型作为实现推理高效部署的可能路径。
英文摘要：
Mixture-of-Experts (MoE) models deliver high quality at low training FLOPs, but this efficiency often vanishes at inference. We identify a double penalty that structurally disadvantages MoE architectures during decoding: first, expert routing fragments microbatches and reduces weight reuse; second, massive resident expert pools reduce high-bandwidth memory (HBM) headroom for the KV cache. This phenomenon, formalized as reuse fragmentation, pushes feed-forward networks (FFNs) into a bandwidth-bound regime, especially at long context lengths. We introduce the $qs$ inequality, a predictive criterion that identifies when MoE is structurally disadvantaged relative to a quality-matched dense model. This criterion unifies sparsity ($s$), the fraction of parameters activated per token, and the quality-equivalence factor ($q$), the size multiplier required for a dense model to match MoE performance. Our evaluation across frontier models including DeepSeek-V3, Qwen3-235B, Grok-1, and Switch-C demonstrates that this fragmentation is a general architectural phenomenon. For DeepSeek-V3 at 128k context, this results in a 4.5x throughput advantage for a quality-matched dense baseline. Crucially, massive architectures like Switch-C can become infeasible on cluster sizes where a quality-matched dense model remains viable. Our results suggest that training-time FLOP efficiency is an incomplete proxy for inference-time performance in long-context serving. They also indicate that MoE may be best viewed as a training-time optimization, with distillation into dense models as a possible path toward inference-efficient deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
