StageFrontier: 面向分布式ML训练的同步感知阶段记账
StageFrontier: Synchronization-Aware Stage Accounting for Distributed ML Training
AI总结：
针对分布式训练中同步掩盖故障根因的问题，提出轻量级在线信号StageFrontier，通过粗粒度阶段时长向量实现精确的暴露时间记账，定位慢节点和阶段，开销低于0.2%。
摘要：
当分布式训练任务变慢时，难点在于知道该查看何处。同步掩盖了原因：一个rank上的停滞表现为其他rank上的等待，因此单个rank上的数据延迟可能表现为整个组的反向时间。始终运行的廉价仪表板（每阶段平均值和最大值）会误读此情况，重复计算相同的暴露延迟或将慢rank淹没在平均值中，而完整的性能分析器虽然能清晰看到问题，但过于沉重无法常开。StageFrontier是一种常开信号，填补了这一空白。每个rank仅报告一个由粗粒度阶段持续时间（数据、前向、反向等）组成的短有序向量，使用CPU挂钟计时，无需同步时钟和内核跟踪。在每个阶段边界，StageFrontier取最领先rank的累积时间；该前沿的增量构成了步骤暴露时间的精确可加性记账，并指向组可见延迟首次出现的阶段和rank，告诉操作员应将重型性能分析器指向何处，而非做出何种修复。该记账是精确的，但仅凭粗粒度信号无法判断领先阶段是真正导致减速还是仅仅与之并行发生；StageFrontier标记需要更多证据来区分这两种情况的窗口，而非进行猜测。一个PyTorch实现在Gloo和NCCL上通过128个rank增加了不到0.2%的吞吐量开销，在隐藏rank的DDP测试的所有50行中，将注入的故障排在前两位嫌疑中，并且一旦将其跟踪缩减为相同的粗粒度阶段，即可从0.11 MB的摘要（而非15.81 GB的跟踪）中恢复出与PyTorch Profiler、HTA和Nsight Systems相同的顶级阶段路由。
英文摘要：
When a distributed training job slows down, the hard part is knowing where to look. Synchronization hides the cause: a stall on one rank shows up as a wait on the others, so a data delay on a single rank can surface as backward time across the group. The cheap dashboards that run all the time -- per-stage averages and maxima -- misread this, double-counting the same exposed delay or burying the slow rank in an average, while full profilers see it clearly but are far too heavy to leave on. StageFrontier is an always-on signal that closes this gap. Each rank reports only a short ordered vector of coarse stage durations -- data, forward, backward, and so on -- timed with CPU wall-clock, with no synchronized clocks and no kernel tracing. At each stage boundary, StageFrontier takes the cumulative time of whichever rank is furthest along; the increments of this frontier form an exact, additive accounting of the step's exposed time and point to the stage and rank where group-visible delay first appears, telling an operator where to aim a heavy profiler, not which fix to make. The accounting is exact, but the coarse signal alone cannot tell whether a leading stage truly caused the slowdown or merely ran alongside it; StageFrontier labels the windows where that distinction needs more evidence instead of guessing. A PyTorch implementation adds under 0.2% throughput overhead through 128 ranks on Gloo and NCCL, places injected faults among its top two suspects on all 50 rows of a hidden-rank DDP test, and recovers the same top-stage routing as PyTorch Profiler, HTA, and Nsight Systems once their traces are reduced to the same coarse stages -- from a 0.11 MB summary instead of a 15.81 GB trace.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
