Mixture-of-Experts (MoE) models facilitate edge deployment by decoupling model capacity from active computation, yet their large memory footprint drives the need for GPU systems with near-data processing (NDP) capabilities that offload experts to dedicated processing units. However, deploying MoE models on such edge-based GPU-NDP systems faces three critical challenges: 1) severe load imbalance across NDP units due to non-uniform expert selection and expert parallelism, 2) insufficient GPU utilization during expert computation within NDP units, and 3) extensive data pre-profiling necessitated by unpredictable expert activation patterns for pre-fetching. To address these challenges, this paper proposes an efficient inference framework featuring three key optimizations. First, the underexplored tensor parallelism in MoE inference is exploited to partition and compute large expert parameters across multiple NDP units simultaneously towards edge low-batch scenarios. Second, a load-balancing-aware scheduling algorithm distributes expert computations across NDP units and GPU to maximize resource utilization. Third, a dataset-free pre-fetching strategy proactively loads frequently accessed experts to minimize activation delays. Experimental results show that our framework enables GPU-NDP systems to achieve 2.41x on average and up to 2.56x speedup in end-to-end latency compared to state-of-the-art approaches, significantly enhancing MoE inference efficiency in resource-constrained environments.
TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China(SKLP,计算技术研究所,中国科学院,北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京)
Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Hong Kong Baptist University(香港 Baptist 大学);The Hong Kong University of Science(香港科学与技术大学)
ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
School of Electronic Science and Engineering, Nanjing University, China.(南京大学电子科学与工程学院);National Key Laboratory for Novel Software Technology, Nanjing University, China.(南京大学新型软件技术国家重点实验室)
Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling
AI Chip Center for Emerging Smart Systems(新兴智能系统芯片中心);The Hong Kong University of Science and Technology(香港科技大学);Shanghai UniVista Industrial Software Group Co., Ltd.(上海华大九天工业软件集团有限公司)
ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
CFAR, Agency for Science, Technology and Research (A*STAR), Singapore(新加坡科技研究局CFAR);The Hong Kong University of Science and Technology, Hong Kong(香港科技大学);Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));Hong Kong Baptist University, Hong Kong(香港浸会大学);Nanyang Technological University, Singapore(南洋理工大学);The Hong Kong University of Science and Technology (Guangzhou), China(香港科技大学(广州))
HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs
University of Science and Technology of China(中国科学技术大学);Huawei Technologies Co., Ltd(华为技术有限公司);Peking University(北京大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training
Zhejiang University(浙江大学);Institute for Communication Systems (ICS), University of Surrey(萨里大学通信系统研究所);Xi’an Jiaotong-Liverpool University(西交利物浦大学);Northeastern University(东北大学);Khalifa University(哈利法大学)
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Integrated Circuits, Peking University(北京大学集成电路学院);Institute of Electronic Design Automation, Peking University(北京大学电子设计自动化研究院);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路高精尖创新中心);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
University of Chinese Academy of Sciences(中国科学院大学);Huawei Technologies Co. Ltd(华为技术有限公司);Huawei Ascend Computing Technology Development(华为昇腾计算技术发展中心);Shanghai Jiao Tong University(上海交通大学)
Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
Hong Kong University of Science and Technology(香港科技大学);Shanghai Jiao Tong University(上海交通大学);Huawei Technologies Co. Ltd.(华为技术有限公司);Peking University(北京大学);Shanghai Qi Zhi Institute(上海人工智能实验室)
RailWave: Adaptive Spatial and Temporal Scheduling for Expert-Parallel Communication
Sun Yat-sen University(中山大学);Southeast University(东南大学);Monash University(蒙纳士大学);National University of Singapore(新加坡国立大学);Shenzhen University of Advanced Technology(深圳先进技术研究院);Renmin University of China(中国人民大学)
BrownoutMoE: Learning Structure-Aware Expert Grouping for Adaptive Mixture-of-Experts Model Serving
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Victoria University of Wellington(惠灵顿维多利亚大学);State Key Lab of IoTSC, University of Macau(澳门大学物联网与智能计算国家重点实验室)
RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training
Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology(深圳先进技术学院计算机科学与控制工程系);Institute for Network Sciences and Cyberspace, Tsinghua University(清华大学网络科学与网络空间研究院);Department of Information Engineering, Chinese University of Hong Kong(香港中文大学信息工程系);China Mobile(中国移动)