加速混合专家推理的自适应专家分割机制
Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
AI总结：
针对MoE推理中专家缓存命中率低和加载延迟大的问题，提出MoEpic系统，通过垂直分割专家并缓存顶部段，结合自适应缓存配置算法，节省约一半GPU成本并降低推理延迟37.51%-65.73%。
摘要：
混合专家（Mixture-of-Experts, MoE）已成为现代大型语言模型（LLMs）的一种有前景的架构。然而，庞大的参数对GPU内存（即VRAM）提出了沉重需求，阻碍了MoE LLMs的广泛采用。将专家参数卸载到CPU RAM中，为缓解MoE推理的VRAM需求提供了一种有效途径。现有方法通常在VRAM中缓存一小部分专家，并在推理过程中从RAM动态预取专家，但由于缓存命中率低和专家加载延迟大，导致推理速度显著下降。在这项工作中，我们提出了MoEpic，一种具有新颖专家分割机制的高效MoE推理系统。具体而言，每个专家被垂直划分为两个段：顶部段和底部段。MoEpic缓存热门专家的顶部段，从而在有限的VRAM预算下存储更多专家，进而提高缓存命中率。在每一层的推理过程中，MoEpic预测并预取下一层被激活的专家。由于缓存专家的顶部段无需获取，加载时间得以减少，这使得高效的传输-计算重叠成为可能。然而，MoEpic的性能关键取决于缓存配置（即每层的VRAM预算和专家分割比例）。为此，我们提出了一种基于定点迭代的分治算法，用于自适应缓存配置。在流行的MoE LLMs上进行的大量实验表明，与基线相比，MoEpic可以节省约一半的GPU成本，同时将推理延迟降低约37.51%-65.73%。
英文摘要：
Mixture-of-Experts (MoE) has emerged as a promising architecture for modern large language models (LLMs). However, massive parameters impose heavy GPU memory (i.e., VRAM) demands, hindering the widespread adoption of MoE LLMs. Offloading the expert parameters to CPU RAM offers an effective way to alleviate the VRAM requirements for MoE inference. Existing approaches typically cache a small subset of experts in VRAM and dynamically prefetch experts from RAM during inference, leading to significant degradation in inference speed due to the poor cache hit rate and substantial expert loading latency. In this work, we propose MoEpic, an efficient MoE inference system with a novel expert split mechanism. Specifically, each expert is vertically divided into two segments: top and bottom. MoEpic caches the top segment of hot experts, so that more experts will be stored under the limited VRAM budget, thereby improving the cache hit rate. During each layer's inference, MoEpic predicts and prefetches the activated experts for the next layer. Since the top segments of cached experts are exempt from fetching, the loading time is reduced, which allows efficient transfer-computation overlap. Nevertheless, the performance of MoEpic critically depends on the cache configuration (i.e., each layer's VRAM budget and expert split ratio). To this end, we propose a divide-and-conquer algorithm based on fixed-point iteration for adaptive cache configuration. Extensive experiments on popular MoE LLMs demonstrate that MoEpic can save about half of the GPU cost, while lowering the inference latency by about 37.51%-65.73% compared to the baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
