arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.01433cs.LGcs.DC
收藏

HOBBIT:用于快速MoE推理的混合精度专家卸载系统

HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference

Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)

Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng-Ann Heng, Chao Li, Minyi Guo

AI总结:

HOBBIT通过混合精度专家卸载、动态加载、自适应预取和多维缓存,在边缘设备上实现高达9.93倍的MoE推理解码加速,同时保持模型精度。

摘要:

混合专家(MoE)架构在大语言模型(LLM)时代展现出显著优势,以更低的推理成本提供增强的能力。然而,由于巨大的内存需求,在内存受限的边缘设备上部署基于MoE的大语言模型仍然具有挑战性。现有的专家卸载方法虽然缓解了内存需求,但往往会产生显著的专家加载成本或损害模型精度。我们提出了HOBBIT,一个混合精度专家卸载系统,以实现灵活高效的MoE推理。我们的关键洞察是,动态地用低精度版本替换不太重要的缓存未命中专家,可以大幅减少专家加载延迟,同时保持模型精度。HOBBIT引入了三种创新技术,映射了MoE计算的自然层次结构:(1)令牌级动态专家加载机制,(2)层级自适应专家预取技术,以及(3)序列级多维专家缓存策略。这些创新充分利用了混合精度专家推理的优势。通过在著名的LLM推理框架Llama.cpp上实现HOBBIT,我们在不同的边缘设备上使用具有代表性的MoE模型评估其性能。结果表明,与最先进的MoE卸载系统相比,HOBBIT在解码方面实现了高达9.93倍的加速。

英文摘要:

The Mixture-of-Experts (MoE) architecture has demonstrated significant advantages in the era of Large Language Models (LLMs), offering enhanced capabilities with reduced inference costs. However, deploying MoE-based LLMs on memoryconstrained edge devices remains challenging due to their substantial memory requirements. While existing expertoffloading methods alleviate the memory requirements, they often incur significant expert-loading costs or compromise model accuracy. We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency while preserving model accuracy. HOBBIT introduces three innovative techniques that map the natural hierarchy of MoE computation: (1) a token-level dynamic expert loading mechanism, (2) a layer-level adaptive expert prefetching technique, and (3) a sequence-level multidimensional expert caching policy. These innovations fully leverage the benefits of mixedprecision expert inference. By implementing HOBBIT on top of the renowned LLM inference framework Llama.cpp, we evaluate its performance across different edge devices with representative MoE models. The results demonstrate that HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑