通过低成本微分优化分布式训练系统的频繁检查点
Optimizing Frequent Checkpointing via Low-Cost Differential for Distributed Training Systems
AI总结：
通过低成本微分优化分布式训练系统的频繁检查点，减少训练时间达89.2%。
摘要：
大规模深度学习模型的分布式训练常常导致失败，因此检查点通常用于恢复。最新研究集中在频繁检查点以实现快速故障恢复。然而，这会产生大量检查点，带来显著成本，从而降低训练性能。最近，微分检查点被提出以减少成本，但其仅限于推荐系统，因此在通用分布式训练系统中的应用仍待探索。我们提出了\sysname，一个高效的频繁检查点框架，该框架通过重用压缩梯度作为微分检查点以减少成本。此外，\sysname结合了批量梯度写优化以高效地将这些微分保存到存储中。它还动态调整检查点频率和批处理大小以最大化性能。为了在非压缩场景下增强\sysname，我们进一步提出了\sysnameplus，它结合了层间重用快照策略以及增量合并保存策略。在各种工作负载上的实验表明，\sysname和\sysnameplus在检查点频率高达每迭代时，可以将训练时间减少高达89.2%和81.2%。
英文摘要：
Distributed training of large deep-learning models often leads to failures, so checkpointing is commonly employed for recovery. State-of-the-art studies focus on frequent checkpointing for fast recovery from failures. However, frequent checkpointing generates numerous checkpoints, incurring substantial costs and thus degrading training performance. Recently, differential checkpointing has been proposed to reduce costs, but it is limited to recommendation systems, so its application to general distributed training systems remains unexplored. In this paper, we find that gradients generated during distributed training can be reused to construct differential checkpoints, while the former's size is smaller than the latter's, motivating us to reuse gradients for low-cost differential checkpointing. Based on this main idea, we propose \sysname, a frequent checkpointing framework for compression-enabled training systems that reuses compressed gradients as differential checkpoints, eliminating redundant differential computation and reducing checkpoint transmission cost. Furthermore, we extend gradient reuse to scenarios without gradient compression and propose \sysnameplus, which employs layer-wise-reuse snapshotting and incremental-merging persistence to overlap checkpointing with training execution. Experiments on diverse workloads, including billion-parameter-scale models, demonstrate that \sysname and \sysnameplus significantly reduce checkpointing overhead and enable checkpointing at frequencies as high as once per iteration, reducing training time by up to 89.2\% and 81.2\%, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
