FCDP:完全缓存的数据并行用于避免通信的大规模训练
FCDP: Fully Cached Data Parallel for Communication-Avoiding Large-Scale Training
AI总结:
FCDP通过利用主机内存作为快速缓存层,减少节点间通信,提升大规模训练吞吐量。
摘要:
训练十亿参数模型需要将模型状态分布在多个GPU上,使用完全分片数据并行(即ZeRO-3)。虽然ZeRO-3在具有高带宽NVLink和InfiniBand互连的集群上表现良好,但使用普通硬件的研究人员面临严重的节点间全部收集瓶颈。现有的优化方法采取了两种方法:GPU内存缓存(MiCS,ZeRO++)以牺牲内存容量换取减少通信,导致在大模型上出现内存不足失败;主机内存卸载(ZeRO-Offload,ZeRO-Infinity)扩展了容量,但因PCIe开销而降级了吞吐量。我们发现,在带宽受限的集群中,主机内存可以不仅仅作为溢出层,而是一个快速缓存层,其性能优于节点间通信。基于这一见解,我们提出了FCDP,它消除了冗余的节点间通信,同时保持ZeRO-3的最小GPU内存占用。FCDP在主机内存中缓存前向传递参数,并在反向传递过程中通过快速的节点内全部收集进行重用,从而将节点间全部收集减少了50%。对于参数高效微调(PEFT),FCDP只选择性地通信可训练参数以最大化缓存,将节点间流量减少超过99%。在我们的普通集群设置中,FCDP的吞吐量比ZeRO-3高100倍,比ZeRO++高51倍,同时保持ZeRO-3的最大批次大小。
英文摘要:
Training billion-parameter models requires distributing model states across GPUs using fully sharded data parallel (i.e., ZeRO-3). While ZeRO-3 succeeds on clusters with high-bandwidth NVLink and InfiniBand interconnects, researchers with commodity hardware face severe inter-node all-gather bottlenecks. Existing optimizations take two approaches: GPU memory caching (MiCS, ZeRO++) trades memory capacity for reduced communication, triggering out-of-memory failures on large models; host memory offloading (ZeRO-Offload, ZeRO-Infinity) extends capacity but degrades throughput due to PCIe overhead. We observe that on bandwidth-limited clusters, host memory can serve not as an overflow tier but as a fast caching layer that outperforms inter-node communication. Based on this insight, we propose FCDP, which eliminates redundant inter-node communication while preserving ZeRO-3's minimal GPU memory footprint. FCDP caches forward-pass parameters in host memory and reuses them during the backward pass via fast intra-node all-gather, reducing inter-node all-gather by 50%. For parameter-efficient fine-tuning (PEFT), FCDP selectively communicates only trainable parameters to maximize caching, reducing inter-node traffic by over 99%. In our commodity cluster setup, FCDP achieves up to 100x higher throughput than ZeRO-3 and 51x higher than ZeRO++, while maintaining ZeRO-3's maximum batch size.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。