AI 中文总结
本文研究 GPU 扩展域中同步代价的来源，发现其主要源于 GEMM 内核执行时间的跨秩差异，且该代价随域规模增长，颠覆了互连带宽扩展的主流认知。
AI 中文摘要
GPU 扩展域已成为现代机器学习基础设施的核心组成部分，其设计遵循互连带宽和域规模呈指数级增长的明确轨迹。本文指出这两种趋势存在矛盾。通过对四个语言模型和三款最新 GPU 架构中数十万次集合操作的研究，我们发现，尽管在统一 fabrics 上的相同硬件上执行相同内核，扩展域内的 GPU 在集合屏障处的到达时间仍相差数百至数千微秒。我们将这种等待时间称为同步代价，并表明在 8-GPU 扩展域中，它可消耗超过 50%的集合通信时间。为理解该代价的来源，我们设计了一种基于图的算法，该算法基于每个秩的内核迹线运行，揭示 GEMM 内核执行时间的跨秩差异占该开销的 78%。我们应用极值理论对这种差异进行建模，并证明同步代价随域规模增长。将该模型融入增强型 Hockney 通信代价模型后，我们表明同步代价从根本上限制了带宽扩展的回报，并颠覆了关于互连带宽应如何随域规模扩展的主流观点。
英文摘要
GPU scale-up domains have become the building block of modern machine learning infrastructure, and their design follows a clear trajectory of exponential growth in both interconnect bandwidth and domain size. This paper argues that these two trends are in tension. Through a study of several hundred thousand collective operations across four language models and three recent GPU architectures, we find that GPUs within a scale-up domain arrive at collective barriers hundreds to thousands of microseconds apart, despite executing identical kernels on identical hardware over a uniform fabric. We call this waiting time the synchronization tax and show that it can consume over 50% of collective communication time in an 8-GPU scale-up domain. To understand the sources of this tax, we design a graph-based algorithm that operates on per-rank kernel traces, revealing that cross-rank variation in GEMM kernel execution times accounts for 78% of this overhead. We apply extreme value theory to model this variation and demonstrate that the synchronization tax grows with domain size. Folding this model into an augmented Hockney communication cost model, we show that the synchronization tax fundamentally limits the return on bandwidth scaling and inverts prevailing beliefs about how interconnect bandwidth should scale with domain size.
