MoEQuant：通过专家均衡采样与亲和度引导增强混合专家大语言模型的量化
MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
AI总结：
MoEQuant针对MoE大语言模型量化中的专家间与专家内不均衡问题，提出专家均衡自采样和亲和度引导量化两种技术，在4-bit量化下显著提升模型精度与效率。
摘要：
混合专家（MoE）大语言模型（LLMs）利用动态路由和稀疏激活来提升效率与可扩展性，在降低计算成本的同时取得了更高的性能。然而，这些模型面临显著的内存开销，限制了其实际部署和更广泛的采用。训练后量化（PTQ）作为一种广泛使用的LLM压缩方法，在应用于MoE模型时会遭遇严重的精度下降和泛化性能减弱。本文研究了MoE的稀疏和动态特性对量化的影响，并识别出两个主要挑战：（1）专家间不均衡，指样本在不同专家之间的分布不均匀，导致对使用频率较低的专家校准不足且存在偏差；（2）专家内不均衡，源于MoE独特的聚合机制，导致不同样本与其被分配专家之间的相关程度各不相同。为应对这些挑战，我们提出了MoEQuant，一个专为MoE LLMs量身定制的新型量化框架。MoEQuant包含两项新技术：1）专家均衡自采样（EBSS）是一种高效采样方法，通过利用token的累积概率和专家均衡指标作为引导因素，高效构建具有均衡专家分布的校准集。2）亲和度引导量化（AGQ），将专家与样本之间的亲和度纳入量化过程，从而准确评估单个样本对MoE层内不同专家的影响。实验表明，MoEQuant取得了显著的性能提升（在4-bit量化下，DeepSeekMoE-16B在HumanEval上的准确率提升超过10个点），并提高了效率。
英文摘要：
Mixture-of-Experts (MoE) large language models (LLMs), which leverage dynamic routing and sparse activation to enhance efficiency and scalability, have achieved higher performance while reducing computational costs. However, these models face significant memory overheads, limiting their practical deployment and broader adoption. Post-training quantization (PTQ), a widely used method for compressing LLMs, encounters severe accuracy degradation and diminished generalization performance when applied to MoE models. This paper investigates the impact of MoE's sparse and dynamic characteristics on quantization and identifies two primary challenges: (1) Inter-expert imbalance, referring to the uneven distribution of samples across experts, which leads to insufficient and biased calibration for less frequently utilized experts; (2) Intra-expert imbalance, arising from MoE's unique aggregation mechanism, which leads to varying degrees of correlation between different samples and their assigned experts. To address these challenges, we propose MoEQuant, a novel quantization framework tailored for MoE LLMs. MoE-Quant includes two novel techniques: 1) Expert-Balanced Self-Sampling (EBSS) is an efficient sampling method that efficiently constructs a calibration set with balanced expert distributions by leveraging the cumulative probabilities of tokens and expert balance metrics as guiding factors. 2) Affinity-Guided Quantization (AGQ), which incorporates affinities between experts and samples into the quantization process, thereby accurately assessing the impact of individual samples on different experts within the MoE layer. Experiments demonstrate that MoEQuant achieves substantial performance gains (more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization) and boosts efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
