基于上下文的混合专家推理在支持CXL的GPU-NDP系统中
Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems
AI总结：
本文提出一种基于上下文的混合专家推理系统，利用CXL-NDP减少内存传输成本，提升解码吞吐量8.7倍，精度仅下降0.13%。
摘要：
混合专家（MoE）模型通过条件计算扩展大语言模型，但一旦专家权重超过GPU内存容量，推理过程就会变得内存受限。在此情况下，权重必须卸载到外部内存，而获取它们会带来高昂且重复的传输成本。我们通过采用附带CXL的近数据处理（CXL-NDP）作为卸载层级来执行冷专家，将昂贵的参数移动转化为更便宜的激活移动。与以往的GPU-NDP系统相比，这些系统大多缺乏上下文意识且反应式，我们开发了一种上下文感知的MoE系统，利用预填充阶段的激活统计信息来指导解码阶段的专家放置，动态地将热专家固定在GPU侧的HBM上，并将剩余部分映射到CXL-NDP。为了满足NDP的有限计算吞吐量，我们引入了上下文感知的混合精度量化，根据预填充阶段为每个专家分配位宽（1-4位）。最终的MoE推理系统在GPU和NDP之间重叠执行，同时最小化跨设备移动。在GPU-NDP系统上的评估显示，我们的方法在解码吞吐量上比最先进的方法提高了8.7倍，同时仅导致平均精度下降0.13%。
英文摘要：
Mixture-of-Experts (MoE) models scale large language models through conditional computation, but inference becomes memory-bound once expert weights exceed the capacity of GPU memory. In this case, weights must be offloaded to external memory, and fetching them incurs costly and repeated transfers. We address this by adopting CXL-attached near-data processing (CXL-NDP) as the offloading tier to execute cold experts in place, converting expensive parameter movement into cheaper activation movement. Unlike prior GPU-NDP systems that are largely context-agnostic and reactive, we develop a context-aware MoE system that uses prefill-stage activation statistics to guide decoding-stage expert placement, dynamically pins hot experts in GPU-side HBM, and maps the remainder to CXL-NDP. To meet NDP's limited compute throughput, we introduce context-aware mixed-precision quantization that allocates per-expert bitwidths (1-4 bit) based on prefill stage. The resulting MoE inference system overlaps GPU and NDP execution while minimizing cross-device movement. The evaluation on the GPU-NDP system shows that our approach achieves up to an 8.7-fold decoding throughput improvement over the state-of-the-art method, while incurring only a 0.13% average accuracy drop.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
