用于稀疏专家混合模型LLM插拔式负载平衡的复制和量化策略
A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
AI总结：
提出Replicate-and-Quantize策略，通过动态工作量重新平衡提升稀疏混合专家模型LLM的推理效率和稳定性。
摘要：
稀疏混合专家（SMoE）架构正被越来越多地用于高效扩展大型语言模型，能够在固定计算预算下提供强大的准确性。然而，SMoE模型常常在专家之间出现严重的负载不平衡，其中一小部分专家接收大部分令牌，而其他专家则被低估利用。以往的研究主要集中在训练时间的解决方案，如路由正则化或辅助损失，使得部署关键的推理时间行为研究较少。我们对推理期间的专家路由进行了系统分析，并识别出三个发现：（i）随着批量大小的增加，负载不平衡持续存在并恶化；（ii）选择频率并不可靠地反映专家的重要性；（iii）整体专家工作量和重要性可以通过一个小的校准集进行估计。这些见解促使了在不重新训练或修改路由器的情况下，通过推理时间机制重新平衡工作量的机制。我们提出了复制和量化（R&Q），一种训练免费且近无损的动态工作量重新平衡框架。在每一层中，重型击专家被复制以增加并行容量，而较不关键的专家和复制体被量化以保持在原始内存预算内。我们还引入了负载不平衡分数（LIS）来通过比较重型击负载与等分配基线来衡量路由偏斜。在代表性SMoE模型和基准测试中的实验显示，负载不平衡减少了1.4倍，同时准确性保持在±0.6%以内，使推理更加可预测和高效。
英文摘要：
Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently, delivering strong accuracy under fixed compute budgets. However, SMoE models often suffer from severe load imbalance across experts, where a small subset of experts receives most tokens while others are underutilized. Prior work has focused mainly on training-time solutions such as routing regularization or auxiliary losses, leaving inference-time behavior, which is critical for deployment, less explored. We present a systematic analysis of expert routing during inference and identify three findings: (i) load imbalance persists and worsens with larger batch sizes, (ii) selection frequency does not reliably reflect expert importance, and (iii) overall expert workload and importance can be estimated using a small calibration set. These insights motivate inference-time mechanisms that rebalance workloads without retraining or router modification. We propose Replicate-and-Quantize (R&Q), a training-free and near-lossless framework for dynamic workload rebalancing. In each layer, heavy-hitter experts are replicated to increase parallel capacity, while less critical experts and replicas are quantized to remain within the original memory budget. We also introduce a Load-Imbalance Score (LIS) to measure routing skew by comparing heavy-hitter load to an equal allocation baseline. Experiments across representative SMoE models and benchmarks show up to 1.4x reduction in imbalance with accuracy maintained within +/-0.6%, enabling more predictable and efficient inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
