Mixture-of-Experts (MoE) architectures are widely used in modern large language models and multimodal models. However, inference efficiency is often limited by highly dynamic and skewed expert workloads across different modalities. During the prefill stage with large batch sizes, vision tokens frequently dominate the input sequences. Under expert parallelism (EP), this leads to severe load imbalance, where a subset of devices becomes overloaded, reducing overall system throughput. We propose ReaLB, a real-time load balancing method for multimodal MoE (MMoE) inference that introduces zero scheduling overhead. ReaLB dynamically adjusts the computation precision of MoE experts at runtime on a per-EP-rank basis. For ranks dominated by vision-heavy experts, ReaLB assigns lower-precision computation to improve execution efficiency by exploiting FP4 Tensor Cores. ReaLB does not require redundant experts or additional memory allocation. Instead, it performs layer-wise expert precision transformation on the fly and hides the associated overhead within the dispatch phase before MoE computation. Experiments on representative MMoE models show that ReaLB achieves 1.10$\times$-1.32$\times$ end-to-end speedup while limiting average accuracy degradation to within 1%.
RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training
Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology(深圳先进技术学院计算机科学与控制工程系);Institute for Network Sciences and Cyberspace, Tsinghua University(清华大学网络科学与网络空间研究院);Department of Information Engineering, Chinese University of Hong Kong(香港中文大学信息工程系);China Mobile(中国移动)
NanoCP: Taming Load Imbalance in Data-Expert Parallel Decoding under Variable-Length Workloads
Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Huazhong University of Science and Technology(华中科技大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);Harbin Institute of Technology(哈尔滨工业大学);Peking University(北京大学);SenseTime(商汤科技)