发表机构
Tsinghua University; BNRist（清华大学; 北京信息科学与技术国家研究中心）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对大规模MoE在异构平台上的部署难题，RapidMoE利用跨不对称性，通过残差分割实现比特级卸载，并采用统一多级重要性仲裁，在解码和预填充阶段分别获得高达3.5倍和2.1倍的加速。
AI 中文摘要
混合专家（MoE）模型的广泛采用催生了对异构平台部署日益增长的需求。然而，这暴露了大规模MoE的算法需求与异构硬件迥异特性之间的根本性不匹配。现有的CPU-GPU混合推理系统未能解决这一问题，因为它们在将专家加载到GPU时要么遭遇PCIe带宽瓶颈，要么过度依赖CPU计算。因此，随着参数规模扩大，这导致资源利用率低下且不可避免地违反固定延迟预算。在本文中，我们识别并利用跨不对称性（Cross-Asymmetry）——即MoE路由的算法工作负载偏斜与异构硬件物理差异之间的结构对齐。为此，我们引入RapidMoE，一种用于高效大规模MoE推理的残差卸载系统。我们提出RapidMoE如何利用残差分割框架实现从专家级到比特级的卸载范式转变，这一转变在三个关键维度展开：（1）数据表示，实现紧凑且解耦的存储；（2）路由策略，将计算划分为与硬件能力对齐的双路径；（3）执行并行性，在设备间调度均衡的存储-计算工作负载。我们进一步采用新颖的统一多级重要性仲裁机制，在运行时自适应调整关键专家集，确保精度-延迟帕累托前沿。这些创新利用了固有的跨不对称性，从根本上打破了算法-硬件错位。实验结果表明，与最先进的（SOTA）卸载系统相比，RapidMoE在解码阶段实现了高达3.5倍的加速，在预填充阶段实现了高达2.1倍的加速。
英文摘要
The widespread adoption of Mixture-of-Experts (MoE) has created a growing need for deployment on heterogeneous platforms. However, it exposes a fundamental mismatch between the algorithmic demands of large-scale MoE and the disparate characteristics of hardware.Existing CPU-GPU hybrid inference systems fail to resolve this as they either encounter PCIe bandwidth bottlenecks when loading experts to GPUs, or rely heavily on CPU computation. Consequently, this leads to low resource utilization and inevitable violations of fixed latency budgets as parameters scale. In this paper, we identify and exploit Cross-Asymmetry--a structural alignment between the algorithmic workload skew of MoE routing and the physical disparity of heterogeneous hardware. To this end, we introduce RapidMoE, a residual offloading system for efficient large-scale MoE inference. We propose how RapidMoE leverages a residual-split framework to enable offloading paradigm shift from expert-level to bit-level, which unfolds across three key dimensions: (1) data representation, enabling compact and decoupled storage; (2) routing strategy, partitioning computation into dual paths aligned with hardware capabilities; (3) execution parallelism, scheduling a balanced storage-compute workload across devices. We further employ a novel Unified Multi-Level Importance Arbitration to adaptively adjust the critical expert set at runtime, ensuring the accuracy-latency Pareto frontier. These innovations exploit inherent cross-asymmetry, fundamentally breaking the algorithm-hardware misalignment. Experimental results show that RapidMoE achieves up to 3.5x speedup in decoding and 2.1x speedup in prefill compared to state-of-the-art (SOTA) offloading systems.
CommentsAccepted by EuroSys 2027. 17 pages
