arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00509cs.DC
收藏

PROBE: 通过实时预测预取在MoE推理中协同平衡计算与通信

PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching

Kuaishou Technology(快手科技)

Qianchao Zhu, Xucheng Ye, Yuliang Liu, Haodong Ouyang, Chengru Song

AI总结:

PROBE通过实时预测预取技术,在MoE推理中平衡计算与通信,减少预填延迟并提升解码吞吐量。

摘要:

混合专家模型已成为通过每个标记仅激活稀疏子集专家来扩展大语言模型的主要架构。然而,对延迟敏感的MoE推理面临根本性的矛盾:虽然专家并行性提高了内存效率,但也加剧了执行延迟问题。在现实世界的服务中,连续批处理和多样化的并发请求导致快速语义变化,导致专家热点在GPU之间突然迁移,触发计算偏斜和网络拥堵的双重惩罚。我们提出了PROBE,一种能够实时平衡计算与通信的推理系统。PROBE引入了连续前瞻流水线,该流水线能够主动预测、规划和预取后续层,同时保持所有控制开销在关键路径之外。PROBE包括:(1)一个基于门控初始化的前瞻预测器,该预测器可以提炼目标路由器以高保真度预测下一层专家激活;(2)一个硬件感知的平衡规划求解器,该求解器在严格的隐藏窗口约束下联合优化动态专家复制和令牌分配;(3)一个相位锁定的协同调度策略,该策略使用分相传输在计算过程中隐藏带宽密集型专家传输,而无需竞争All-to-All收集。实验表明,PROBE将预填延迟减少了高达1.32倍,比最先进的基线提高了解码吞吐量高达1.26倍,特别是在极端工作负载波动下。

英文摘要:

Mixture-of-Experts models have become a dominant architecture for scaling Large Language Models by activating only a sparse subset of experts per token. However, latency-critical MoE inference faces a fundamental tension: while expert parallelism improves memory efficiency, it also amplifies execution stragglers. In real-world serving, continuous batching and diverse concurrent requests induce rapid semantic shifts, causing expert hotspots to migrate abruptly across GPUs and triggering the 'double penalty' of coupled computational skew and network congestion. We propose PROBE, an inference system that co-balances computation and communication in real time. PROBE introduces Continuous Lookahead Pipelining, which proactively predicts, plans, and prefetches for upcoming layers while keeping all control overheads off the critical path. PROBE consists of: (1) a Gate-Initialized Lookahead Predictor that distills the target router to forecast next-layer expert activation with high fidelity; (2) a Hardware-Aware Balance Planning solver that jointly optimizes dynamic expert replication and token assignment under strict hiding-window constraints; and (3) a Phase-Locked Co-Scheduling policy that uses split-phase transmission to hide bandwidth-intensive expert transfers behind computation without contending with All-to-All collectives. Experiments show that PROBE reduces prefill latency by up to 1.32X and improves decoding throughput by up to 1.26X over state-of-the-art baselines, especially under extreme workload volatility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑