SWIFT：面向大规模DNN训练的加速故障恢复
SWIFT: Expedited Failure Recovery for Large-scale DNN Training
AI总结：
本文提出SWIFT，一种用于分布式DNN训练的新型故障恢复设计，通过解决模型状态不一致并利用数据并行副本及基于日志的重计算，在不影响吞吐量和准确性的前提下显著降低恢复开销，总训练时间最高加速1.16倍。
摘要：
随着深度学习模型的规模越来越大，训练需要更长的时间和更多的资源，使得容错性变得越来越关键。现有的最先进方法（如CheckFreq和Elastic Horovod）需要在内存中备份一份模型状态（即参数和优化器状态）的副本，这对于大型模型来说代价高昂，并会导致不可忽视的开销。本文提出了SWIFT，一种用于分布式深度神经网络训练的新型恢复设计，它在不影响训练吞吐量和模型准确性的前提下，显著降低了故障恢复开销。SWIFT不制作模型状态的额外副本，而是解决由故障引起的模型状态不一致性，并利用数据并行中的模型状态副本进行故障恢复。当副本不可用时，我们提出了一种基于日志的方法，记录中间数据并重放计算以在故障发生时恢复丢失的状态。重计算被分布到多台机器上，以进一步加速故障恢复。我们还选择性地记录中间数据，探索恢复时间与中间数据存储开销之间的权衡。评估表明，与最先进的方法相比，SWIFT显著减少了故障恢复时间，并在无故障执行期间实现了相似或更好的训练吞吐量，且未降低最终模型准确性。与最先进的方法相比，SWIFT在总训练时间上还能实现高达1.16倍的加速。
英文摘要：
As the size of deep learning models gets larger and larger, training takes longer time and more resources, making fault tolerance more and more critical. Existing state-of-the-art methods like CheckFreq and Elastic Horovod need to back up a copy of the model state (i.e., parameters and optimizer states) in memory, which is costly for large models and leads to non-trivial overhead. This paper presents SWIFT, a novel recovery design for distributed deep neural network training that significantly reduces the failure recovery overhead without affecting training throughput and model accuracy. Instead of making an additional copy of the model state, SWIFT resolves the inconsistencies of the model state caused by the failure and exploits the replicas of the model state in data parallelism for failure recovery. We propose a logging-based approach when replicas are unavailable, which records intermediate data and replays the computation to recover the lost state upon a failure. The re-computation is distributed across multiple machines to accelerate failure recovery further. We also log intermediate data selectively, exploring the trade-off between recovery time and intermediate data storage overhead. Evaluations show that SWIFT significantly reduces the failure recovery time and achieves similar or better training throughput during failure-free execution compared to state-of-the-art methods without degrading final model accuracy. SWIFT can also achieve up to 1.16x speedup in total training time compared to state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
