The Mixture-of-Experts (MoE) architecture has demonstrated significant advantages in the era of Large Language Models (LLMs), offering enhanced capabilities with reduced inference costs. However, deploying MoE-based LLMs on memoryconstrained edge devices remains challenging due to their substantial memory requirements. While existing expertoffloading methods alleviate the memory requirements, they often incur significant expert-loading costs or compromise model accuracy. We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency while preserving model accuracy. HOBBIT introduces three innovative techniques that map the natural hierarchy of MoE computation: (1) a token-level dynamic expert loading mechanism, (2) a layer-level adaptive expert prefetching technique, and (3) a sequence-level multidimensional expert caching policy. These innovations fully leverage the benefits of mixedprecision expert inference. By implementing HOBBIT on top of the renowned LLM inference framework Llama.cpp, we evaluate its performance across different edge devices with representative MoE models. The results demonstrate that HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
Institute of Automation, CAS(中国科学院自动化研究所);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院);Shanghai Jiao Tong University(上海交通大学);AiRiA
MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
Department of Computing(计算系);Hong Kong Polytechnic University(香港理工大学);Mathematics and Computer Science Division(数学与计算机科学 division);Argonne National Laboratory(阿贡国家实验室);The University of Hong Kong(香港大学);Stevens Institute of Technology(史蒂文斯理工学院);Department of Computer Science and Engineering(计算机科学与工程系);University of California, Merced(加州大学默塞德分校)
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Integrated Circuits, Peking University(北京大学集成电路学院);Institute of Electronic Design Automation, Peking University(北京大学电子设计自动化研究院);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路高精尖创新中心);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
Guangzhou University(广州大学);Institue of Automation Chinese Academy of Sciences(中国科学院自动化研究所);ByteDance Inc.(字节跳动公司);University of Science and Technology Beijing(北京科技大学)
PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts
Department of Engineering, King’s College London(伦敦国王学院工程系);Institute for Digital Communication, Friedrich-Alexander-Universität Erlangen-Nürnberg(埃朗根-纽伦堡弗里德里希-亚历山大大学数字通信研究所)
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
Cognizant AI & Analytics(高知特人工智能与分析部门);Cognizant AI Lab(高知特人工智能实验室);Southern Methodist University (SMU)(南卫理公会大学);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校)
Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems
Rensselaer Polytechnic Institute(罗切斯特理工学院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);IBM Research Europe(IBM欧洲研究院);IBM T. J. Watson Research Center(IBM托马斯·贾诺斯·沃森研究中心)