EAC-MoE：面向混合专家大语言模型的专家选择感知压缩器
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
AI总结：
针对MoE大模型内存占用高和推理加速不足的问题，提出EAC-MoE，通过量化校准路由器与按专家使用频率剪枝，实现低性能损失的压缩与加速。
摘要：
混合专家（MoE）在扩展大语言模型（LLM）方面展现了巨大的潜力。然而，它受到两个关键挑战的阻碍：（1）加载所有专家需要消耗大量GPU内存；（2）低激活参数无法等效转化为推理加速效果。在这项工作中，我们提出了EAC-MoE，一种面向MoE-LLM的专家选择感知压缩器，它从量化和剪枝的角度深入契合MoE的特性，并引入两个模块分别应对这两个挑战：（1）由低比特量化引起的专家选择偏差是导致MoE-LLM性能下降的主要因素。基于此，我们提出了带有专家选择校准的量化（QESC），通过校准MoE内的路由器来缓解专家选择偏差；（2）总有一些专家对相应任务并不关键，却导致推理延迟。因此，我们提出了基于专家选择频率的剪枝（PESF），通过剪枝当前任务中较少使用的专家来显著提升推理速度。大量实验表明，我们的方法在性能下降极小的情况下，显著减少了内存使用并提高了推理速度。
英文摘要：
Mixture-of-Experts (MoE) has demonstrated promising potential in scaling LLMs. However, it is hindered by two critical challenges: (1) substantial GPU memory consumption to load all experts; (2) low activated parameters cannot be equivalently translated into inference acceleration effects. In this work, we propose EAC-MoE, an Expert-Selection Aware Compressor for MoE-LLMs, which deeply aligns with the characteristics of MoE from the perspectives of quantization and pruning, and introduces two modules to address these two challenges respectively: (1) The expert selection bias caused by low-bit quantization is a major factor contributing to the performance degradation in MoE-LLMs. Based on this, we propose Quantization with Expert-Selection Calibration (QESC), which mitigates the expert selection bias by calibrating the routers within the MoE; (2) There are always certain experts that are not crucial for the corresponding tasks, yet causing inference latency. Therefore, we propose Pruning based on Expert-Selection Frequency (PESF), which significantly improves inference speed by pruning less frequently used experts for current task. Extensive experiments demonstrate that our approach significantly reduces memory usage and improves inference speed with minimal performance degradation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
