arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.00686cs.DC
收藏

消除多节点巨内核通信中的隐藏序列化

Eliminating Hidden Serialization in Multi-Node Megakernel Communication

Cornell University(康奈尔大学)

Byungsoo Oh, Rachee Singh

AI总结:

本文研究多节点巨内核通信中的隐藏序列化问题,提出Perseus方法通过解耦信号和NIC侧顺序技术,减少序列化开销,提升端到端性能。

摘要:

近期针对混合专家(MoE)推理的巨内核设计将专家计算与细粒度GPU发起的通信融合为单一持久GPU内核,在单节点上优于基于集体的MoE。但这种优势无法直接扩展到多节点推理,因为专家跨越多个通过RDMA织物连接的节点。通信受限的MoE模型在8个节点上退化达10倍,且节点数量增加退化更严重。我们追溯这种退化到基于代理的RDMA传输中的隐藏序列化。每个瓷砖传输与其完成信号之间的顺序要求强制一个阻止NIC流水线的fence,其成本随着并发传输数量增加而增长。因此,计算每个专家太小以至于无法吸收这种膨胀的网络延迟的模型会暴露通信在关键路径上。我们提出了Perseus,通过两种技术消除这种序列化。解耦信号在每目的地粒度上批处理fence,将fence数量减少8倍。NIC侧顺序用硬件fence标志取代代理停滞,因此代理从不阻塞。在基于代理的传输上,Perseus实现高达10.3倍的端到端加速。Perseus在IBRC上匹配或超过IBGDA GPU-direct的性能,表明序列化而非代理基于和GPU-direct传输之间的选择是多节点巨内核性能的瓶颈。

英文摘要:

Recent megakernel designs for Mixture-of-Experts (MoE) inference fuse expert computation with fine-grained, GPU-initiated communication into a single persistent GPU kernel, and outperform collective-based MoE on a single node by overlapping data transfer with compute at tile granularity. This benefit does not carry over cleanly to multi-node inference, where experts span many nodes connected by an RDMA fabric. Communication-bound MoE models regress by up to $10\times$ on 8 nodes, and the regression worsens with node count. We trace this regression to hidden serialization in proxy-based RDMA transports. The ordering requirement between each tile transfer and its completion signal forces a fence that drains the NIC pipeline, and its cost grows with the number of concurrent transfers. As a result, models whose per-expert compute is too small to absorb this inflated network latency expose communication on the critical path. We present \emph{Perseus}, which eliminates this serialization through two techniques. \emph{Decoupled signaling} batches fences at per-destination granularity, reducing fence count by $8\times$. \emph{NIC-side ordering} replaces proxy stalls with hardware fence flags, so the proxy never blocks. On proxy-based transports, Perseus achieves up to 10.3$\times$ end-to-end speedup. Perseus on IBRC matches or exceeds IBGDA GPU-direct by up to 1.2$\times$, which shows that serialization, rather than the choice between proxy-based and GPU-direct transport, is what bounds multi-node megakernel performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑