BuddyMoE：利用专家冗余加速内存受限的混合专家模型推理
BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
AI总结：
针对MoE模型推理时参数超出GPU内存、专家传输延迟及预取失败的问题，BuddyMoE利用专家冗余加速内存受限的MoE推理，旨在平衡推理速度与模型精度。
摘要：
混合专家（MoE）架构通过为每个输入token仅激活专用专家网络的子集来扩展语言模型，从而减少浮点运算次数。然而，现代MoE模型的规模不断增长，导致其完整参数集超出GPU内存容量；例如，Mixtral-8x7B拥有450亿参数，即使每个token仅使用140亿参数，仍需87GB内存。现有系统通过将非活跃专家卸载到CPU内存来缓解这一限制，但跨PCIe互连传输专家会产生显著延迟（约10ms）。预取启发式方法通过预测所需专家来隐藏延迟，但预取失败会导致严重停滞并放大推理延迟。在预取失败的情况下，现有工作提供两种主要解决方案：要么按需获取专家（因PCIe瓶颈导致长时间停滞），要么从计算中丢弃专家（显著降低模型精度）。
英文摘要：
Mixture-of-Experts (MoE) architectures scale language models by activating only a subset of specialized expert networks for each input token, thereby reducing the number of floating-point operations. However, the growing size of modern MoE models causes their full parameter sets to exceed GPU memory capacity; for example, Mixtral-8x7B has 45 billion parameters and requires 87 GB of memory even though only 14 billion parameters are used per token. Existing systems alleviate this limitation by offloading inactive experts to CPU memory, but transferring experts across the PCIe interconnect incurs significant latency (about 10 ms). Prefetching heuristics aim to hide this latency by predicting which experts are needed, but prefetch failures introduce significant stalls and amplify inference latency. In the event of a prefetch failure, prior work offers two primary solutions: either fetch the expert on demand, which incurs a long stall due to the PCIe bottleneck, or drop the expert from the computation, which significantly degrades model accuracy. The critical challenge, therefore, is to maintain both high inference speed and model accuracy when prefetching fails.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
