DALI：一种面向工作负载的高效MoE推理卸载框架
DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
AI总结：
DALI通过动态分配和残差预取技术，提升本地PC上MoE推理效率，实现显著加速。
摘要：
混合专家（MoE）架构显著提升了LLM的容量，而计算量并未成比例增加，但以巨大的参数规模为代价。将MoE专家参数卸载到主机内存，并利用CPU和GPU计算，最近成为在资源受限的本地PC平台上支持此类模型的有前途的方向。尽管有前景，我们注意到现有方法与专家工作负载的动态性不匹配，导致三个根本性的低效：（1）静态专家分配导致严重的CPU-GPU负载不平衡，浪费CPU和GPU资源；（2）现有预取技术无法准确预测高负载专家，导致成本高昂且不准确的预取；（3）GPU缓存策略忽视工作负载动态性，导致命中率低且效果有限。为了解决这些挑战，我们提出了DALI，一种面向工作负载的高效MoE推理卸载框架。为了充分利用硬件资源，DALI首先通过将分配建模为0-1整数优化问题，并在运行时高效地使用贪心分配策略，动态地将专家分配到CPU或GPU。为提高预取准确性，我们开发了一种基于残差的预取方法，利用层间残差信息准确预测高负载专家。此外，我们引入了一种面向工作负载的缓存替换策略，利用专家激活中的时间相关性来提高GPU缓存效率。通过在各种MoE模型和设置上评估，DALI在预填和解码阶段均显著优于最先进的卸载框架。
英文摘要：
Mixture of Experts (MoE) architectures significantly enhance the capacity of LLMs without proportional increases in computation, but at the cost of a vast parameter size. Offloading MoE expert parameters to host memory and leveraging both CPU and GPU computation has recently emerged as a promising direction to support such models on resourceconstrained local PC platforms. While promising, we notice that existing approaches mismatch the dynamic nature of expert workloads, which leads to three fundamental inefficiencies: (1) Static expert assignment causes severe CPUGPU load imbalance, underutilizing CPU and GPU resources; (2) Existing prefetching techniques fail to accurately predict high-workload experts, leading to costly inaccurate prefetches; (3) GPU cache policies neglect workload dynamics, resulting in poor hit rates and limited effectiveness. To address these challenges, we propose DALI, a workloaDAware offLoadIng framework for efficient MoE inference on local PCs. To fully utilize hardware resources, DALI first dynamically assigns experts to CPU or GPU by modeling assignment as a 0-1 integer optimization problem and solving it efficiently using a Greedy Assignment strategy at runtime. To improve prefetching accuracy, we develop a Residual-Based Prefetching method leveraging inter-layer residual information to accurately predict high-workload experts. Additionally, we introduce a Workload-Aware Cache Replacement policy that exploits temporal correlation in expert activations to improve GPU cache efficiency. By evaluating across various MoE models and settings, DALI achieves significant speedups in the both prefill and decoding phases over the state-of-the-art offloading frameworks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
