尽管同构GPU基于深度学习(DL)训练非常流行,但在这种情况下,stragglers的普遍性、原因和影响,以及现有straggler缓解方法的有效性仍然不为人所知,这主要是由于对这些问题的研究有限。为了填补这一空白,我们进行了全面的实验,发现stragglers仍然普遍存在,这是由于CPU和带宽使用不平衡所致。此外,现有的缓解方法将同步随机梯度下降(SSGD)切换到异步SGD(ASGD)可能不会提高时间到准确度(TTA),甚至可能由于其更高的资源消耗而生成更多的stragglers。为了解决这些新发现的问题,我们提出了Straggler Tolerant And Resilient DL训练系统(STAR)。STAR包括新的同步模式,为每个参数更新分组工人。它有一个启发式方法和一个机器学习方法,选择最优的同步模式以最小化TTA,并重新分配资源以支持所选模式,同时最小化对共存任务的影响。此外,它通过避免在分配PSs(消耗高CPU和带宽)和梯度传输时过度加载CPU和带宽资源,主动防止stragglers。我们的基于跟踪的评估表明,在AWS上,STAR在PS和所有减少架构中分别比最先进的系统生成48-84%和51-70%更低的TTA,同时保持SSGD的收敛准确度。STAR的代码已开源。
英文摘要:
Despite the popularity of homogeneous GPU-based deep learning (DL) training, the prevalence, causes and impact of stragglers and the effectiveness of existing straggler mitigation approaches are still not well understood in this scenario due to limited research on these questions. To fill this gap, we conducted comprehensive experiments and found that stragglers remain widespread due to CPU and bandwidth usage imbalances. Additionally, existing mitigation methods that switch from synchronous stochastic gradient descent (SSGD) to asynchronous SGD (ASGD) may not improve Time-To-Accuracy (TTA) and can even generate more stragglers due to its higher resource consumption. To address these newly found problems, we propose the Straggler Tolerant And Resilient DL training system (STAR). STAR includes new synchronization modes that group workers for each parameter updating. It has a heuristic and an ML method to choose the optimal synchronization mode for minimizing TTA, and reallocates resources to support the selected mode while minimizing the impact on co-located jobs. Moreover, it proactively prevents stragglers by avoiding overloading the CPU and bandwidth resources in allocating PSs (which consume high CPU and bandwidth) and in gradient transmission. Our trace-driven evaluation on AWS shows that STAR generates 48-84% and 51-70% lower TTA than state-of-the-art systems in the PS and all-reduce architectures, respectively, while maintaining the converged accuracy of SSGD. The code for STAR is open-sourced.
DSAG: A mixed synchronous-asynchronous iterative method for straggler-resilient learning
我们考虑离散弹性学习。在许多以前的作品中,例如,在编码计算文献中,离散被建模为在工作者之间独立且相同地分布的随机延迟。然而,在许多实际场景中,给定的工人可能会在延长的时间段内掉队。我们提出了一个延迟模型,该模型可以捕获这种行为,并通过在Microsoft Azure,Amazon Web Services(AWS)和小型本地集群上收集的跟踪来证实。在此模型的基础上,我们提出了DSAG,一个混合的同步-异步迭代优化方法,基于随机平均梯度(SAG)方法,结合了及时和陈旧的结果。我们还提出了一个动态的负载平衡策略,以进一步减少离散工人的影响。我们评估了DSAG的主成分分析,铸造作为一个有限和优化问题,一个大型基因组数据集,并在AWS上的100名工人组成的集群上进行逻辑回归,并发现DSAG比SAG快约50%,超过两倍的编码计算方法,对于我们考虑的特定场景。
JSAM: Privacy Straggler-Resilient Joint Client Selection and Incentive Mechanism Design in Differentially Private Federated Learning
Department of Information Engineering, The Chinese University of Hong Kong(信息工程系,香港中文大学);School of Science and Engineering, Shenzhen Institute of Artificial Intelligence and Robotics for Society(科学与工程学院,深圳人工智能与机器人研究院);The Chinese University of Hong Kong(香港中文大学)
ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Hong Kong University of Science and Technology(香港科技大学);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong Baptist University(香港浸会大学);A*STAR(新加坡科技研究局);Northeastern University(东北大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));XGRIDS;National University of Singapore(新加坡国立大学)