Mixture-of-Experts models have become a dominant architecture for scaling Large Language Models by activating only a sparse subset of experts per token. However, latency-critical MoE inference faces a fundamental tension: while expert parallelism improves memory efficiency, it also amplifies execution stragglers. In real-world serving, continuous batching and diverse concurrent requests induce rapid semantic shifts, causing expert hotspots to migrate abruptly across GPUs and triggering the 'double penalty' of coupled computational skew and network congestion.
We propose PROBE, an inference system that co-balances computation and communication in real time. PROBE introduces Continuous Lookahead Pipelining, which proactively predicts, plans, and prefetches for upcoming layers while keeping all control overheads off the critical path. PROBE consists of: (1) a Gate-Initialized Lookahead Predictor that distills the target router to forecast next-layer expert activation with high fidelity; (2) a Hardware-Aware Balance Planning solver that jointly optimizes dynamic expert replication and token assignment under strict hiding-window constraints; and (3) a Phase-Locked Co-Scheduling policy that uses split-phase transmission to hide bandwidth-intensive expert transfers behind computation without contending with All-to-All collectives. Experiments show that PROBE reduces prefill latency by up to 1.32X and improves decoding throughput by up to 1.26X over state-of-the-art baselines, especially under extreme workload volatility.
DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
Institute of Automation, CAS(中国科学院自动化研究所);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院);Shanghai Jiao Tong University(上海交通大学);AiRiA
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
University of Chinese Academy of Sciences(中国科学院大学);Huawei Technologies Co. Ltd(华为技术有限公司);Huawei Ascend Computing Technology Development(华为昇腾计算技术发展中心);Shanghai Jiao Tong University(上海交通大学)
RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training
Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology(深圳先进技术学院计算机科学与控制工程系);Institute for Network Sciences and Cyberspace, Tsinghua University(清华大学网络科学与网络空间研究院);Department of Information Engineering, Chinese University of Hong Kong(香港中文大学信息工程系);China Mobile(中国移动)
NanoCP: Taming Load Imbalance in Data-Expert Parallel Decoding under Variable-Length Workloads
Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Huazhong University of Science and Technology(华中科技大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);Harbin Institute of Technology(哈尔滨工业大学);Peking University(北京大学);SenseTime(商汤科技)
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Integrated Circuits, Peking University(北京大学集成电路学院);Institute of Electronic Design Automation, Peking University(北京大学电子设计自动化研究院);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路高精尖创新中心);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)