arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.03992cs.DCcs.AI
收藏

一种用于边缘GPU-NDP系统高效MoE推断的调度框架

A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems

School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);China Mobile Research Institute(中国移动研究院)

Qi Wu, Chao Fang, Jiayuan Chen, Ye Lin, Yueqi Zhang, Yichuan Bai, Yuan Du, Li Du

AI总结:

本文提出一种高效推断框架,通过张量并行、负载均衡调度和数据集无关预取策略,解决边缘GPU-NDP系统中MoE模型推断的负载不平衡、GPU利用率低和预取数据量大问题,实现端到端延迟平均提升2.41倍。

摘要:

混合专家(MoE)模型通过将模型容量与活跃计算解耦,促进了边缘部署,但其较大的内存足迹推动了需要具备近数据处理(NDP)能力的GPU系统,以将专家卸载到专用处理单元。然而,在此类基于边缘的GPU-NDP系统上部署MoE模型面临三个关键挑战:1)由于非均匀的专家选择和并行性,NDP单元之间存在严重的负载不平衡;2)在NDP单元内专家计算期间GPU利用率不足;3)由于不可预测的专家激活模式,需要大量的数据预配置以进行预取。为了解决这些挑战,本文提出了一种高效的推断框架,包含三个关键优化。首先,利用在MoE推断中被低估的张量并行性,将大专家参数在多个NDP单元上同时划分和计算,以适应边缘低批量场景。其次,一种负载平衡感知的调度算法将专家计算分布到NDP单元和GPU,以最大化资源利用率。第三,一种无需数据集的预取策略主动加载频繁访问的专家,以最小化激活延迟。实验结果表明,我们的框架使GPU-NDP系统在端到端延迟方面平均提升2.41倍,最高达2.56倍,显著提高了资源受限环境中的MoE推断效率。

英文摘要:

Mixture-of-Experts (MoE) models facilitate edge deployment by decoupling model capacity from active computation, yet their large memory footprint drives the need for GPU systems with near-data processing (NDP) capabilities that offload experts to dedicated processing units. However, deploying MoE models on such edge-based GPU-NDP systems faces three critical challenges: 1) severe load imbalance across NDP units due to non-uniform expert selection and expert parallelism, 2) insufficient GPU utilization during expert computation within NDP units, and 3) extensive data pre-profiling necessitated by unpredictable expert activation patterns for pre-fetching. To address these challenges, this paper proposes an efficient inference framework featuring three key optimizations. First, the underexplored tensor parallelism in MoE inference is exploited to partition and compute large expert parameters across multiple NDP units simultaneously towards edge low-batch scenarios. Second, a load-balancing-aware scheduling algorithm distributes expert computations across NDP units and GPU to maximize resource utilization. Third, a dataset-free pre-fetching strategy proactively loads frequently accessed experts to minimize activation delays. Experimental results show that our framework enables GPU-NDP systems to achieve 2.41x on average and up to 2.56x speedup in end-to-end latency compared to state-of-the-art approaches, significantly enhancing MoE inference efficiency in resource-constrained environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑