arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.09685cs.DC
收藏

容错且抗延迟的同构GPU深度学习训练

Straggler Tolerant and Resilient DL Training on Homogeneous GPUs

University of Virginia(弗吉尼亚大学)

Zeyu Zhang, Haiying Shen

AI总结:

STAR系统通过优化同步模式和资源分配,有效降低深度学习训练中的延迟问题,提升训练效率和稳定性。

摘要:

尽管同构GPU基于深度学习(DL)训练非常流行,但在这种情况下,stragglers的普遍性、原因和影响,以及现有straggler缓解方法的有效性仍然不为人所知,这主要是由于对这些问题的研究有限。为了填补这一空白,我们进行了全面的实验,发现stragglers仍然普遍存在,这是由于CPU和带宽使用不平衡所致。此外,现有的缓解方法将同步随机梯度下降(SSGD)切换到异步SGD(ASGD)可能不会提高时间到准确度(TTA),甚至可能由于其更高的资源消耗而生成更多的stragglers。为了解决这些新发现的问题,我们提出了Straggler Tolerant And Resilient DL训练系统(STAR)。STAR包括新的同步模式,为每个参数更新分组工人。它有一个启发式方法和一个机器学习方法,选择最优的同步模式以最小化TTA,并重新分配资源以支持所选模式,同时最小化对共存任务的影响。此外,它通过避免在分配PSs(消耗高CPU和带宽)和梯度传输时过度加载CPU和带宽资源,主动防止stragglers。我们的基于跟踪的评估表明,在AWS上,STAR在PS和所有减少架构中分别比最先进的系统生成48-84%和51-70%更低的TTA,同时保持SSGD的收敛准确度。STAR的代码已开源。

英文摘要:

Despite the popularity of homogeneous GPU-based deep learning (DL) training, the prevalence, causes and impact of stragglers and the effectiveness of existing straggler mitigation approaches are still not well understood in this scenario due to limited research on these questions. To fill this gap, we conducted comprehensive experiments and found that stragglers remain widespread due to CPU and bandwidth usage imbalances. Additionally, existing mitigation methods that switch from synchronous stochastic gradient descent (SSGD) to asynchronous SGD (ASGD) may not improve Time-To-Accuracy (TTA) and can even generate more stragglers due to its higher resource consumption. To address these newly found problems, we propose the Straggler Tolerant And Resilient DL training system (STAR). STAR includes new synchronization modes that group workers for each parameter updating. It has a heuristic and an ML method to choose the optimal synchronization mode for minimizing TTA, and reallocates resources to support the selected mode while minimizing the impact on co-located jobs. Moreover, it proactively prevents stragglers by avoiding overloading the CPU and bandwidth resources in allocating PSs (which consume high CPU and bandwidth) and in gradient transmission. Our trace-driven evaluation on AWS shows that STAR generates 48-84% and 51-70% lower TTA than state-of-the-art systems in the PS and all-reduce architectures, respectively, while maintaining the converged accuracy of SSGD. The code for STAR is open-sourced.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑