Application-Level Differential Checkpointing for HPC Applications with Dynamic Datasets
Kai Keller, Leonardo Bautista Gomez
备注：
This project has received funding from the European Unions Seventh Framework Programme (FP7/2007-2013) and the Horizon 2020 (H2020) funding framework under grant agreement no. H2020-FETHPC-754304 (DEEP-EST); and the LEGaTO Project (legato- project.eu), grant agreement No 780681
High-performance computing (HPC) requires resilience techniques such as checkpointing in order to tolerate failures in supercomputers. As the number of nodes and memory in supercomputers keeps on increasing, the size of checkpoint data also increases dramatically, sometimes causing an I/O bottleneck. Differential checkpointing (dCP) aims to minimize the checkpointing overhead by only writing data differences. This is typically implemented at the memory page level, sometimes complemented with hashing algorithms. However, such a technique is unable to cope with dynamic-size datasets. In this work, we present a novel dCP implementation with a new file format that allows fragmentation of protected datasets in order to support dynamic sizes. We identify dirty data blocks using hash algorithms. In order to evaluate the dCP performance, we ported the HPC applications xPic, LULESH 2.0 and Heat2D and analyze them regarding their potential of reducing I/O with dCP and how this data reduction influences the checkpoint performance. In our experiments, we achieve reductions of up to 62% of the checkpoint time.
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
HPC-AI Technology Inc.（潞晨科技）；Fudan University（复旦大学）；National University of Singapore（新加坡国立大学）
近年来，大规模模型在各个领域都表现出了最先进的性能。然而，训练这样的模型需要各种技术来解决诸如GPU之类的设备上的有限计算能力和存储器的问题。一些常用的技术包括流水线并行、张量并行和激活检查点。虽然现有的工作集中在寻找有效的分布式执行计划（Zheng et al. 2022）和激活检查点调度（Herrmann et al. 2019，Beaumont et al. 2021），但还没有提出联合优化这两个计划的方法。此外，提前编译在很大程度上依赖于准确的内存和计算开销估计，这通常是耗时和误导的。现有的训练系统和机器学习流水线或者物理地执行每个操作数，或者利用缩放的输入张量来估计存储器使用。为了解决这些挑战，我们引入了一个系统，可以联合优化分布式执行和梯度检查点计划。此外，我们还提供了一个易于使用的符号分析器，可以以最小的时间成本为任何PyTorch模型生成内存和计算统计数据。我们的方法允许用户在给定的硬件上并行化他们的模型训练，并以最小的代码更改为基础。源代码可在Colossal-AI GitHub或https://github.com/hpcaitech/ColossalAI上公开获取
