arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.13781cs.LGcs.DC
收藏

为可扩展分布式深度学习保持近最优梯度稀疏化成本

Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning

ETRI(韩国电子通信研究院);Ajou University(明知大学)

Daegun Yoon, Sangyoon Oh

AI总结:

针对分布式训练中梯度稀疏化可扩展性差、负载不均和稀疏度控制不足的问题,提出ExDyna方案,通过独占分区、动态拓扑调整和在线阈值缩放,在保持高精度的同时实现近最优稀疏化成本并提升训练速度。

摘要:

通信开销是扩展分布式训练系统的主要障碍。梯度稀疏化是一种潜在的优化方法,可以在不显著损失模型保真度的情况下减少通信量。然而,现有的梯度稀疏化方法由于算法设计效率低下,可扩展性较差,这显著增加了通信开销。特别是,梯度累积和不充分的稀疏性控制方法严重降低了稀疏化性能。此外,由于工作节点之间梯度选择的工作负载不均衡,通信流量急剧增加。为了解决这些挑战,我们提出了一种名为ExDyna的新型梯度稀疏化方案。在ExDyna中,模型的梯度张量由细粒度块组成,连续的块被分组为不重叠的分区。每个工作节点在其独占分配的分区中选择梯度,因此不会发生梯度累积。为了平衡工作节点之间梯度选择的工作负载,ExDyna通过比较相邻分区的工作负载来调整分区拓扑。此外,ExDyna支持在线阈值缩放,可以实时估计准确的梯度选择阈值。因此,无论模型和数据集如何,ExDyna都能在训练期间满足用户要求的稀疏度水平。因此,ExDyna可以通过保持近最优的梯度稀疏化成本来增强分布式训练系统的可扩展性。在实验中,ExDyna在训练速度和稀疏化性能方面优于最先进的稀疏化器,同时实现了高精度。

英文摘要:

Communication overhead is a major obstacle to scaling distributed training systems. Gradient sparsification is a potential optimization approach to reduce the communication volume without significant loss of model fidelity. However, existing gradient sparsification methods have low scalability owing to inefficient design of their algorithms, which raises the communication overhead significantly. In particular, gradient build-up and inadequate sparsity control methods degrade the sparsification performance considerably. Moreover, communication traffic increases drastically owing to workload imbalance of gradient selection between workers. To address these challenges, we propose a novel gradient sparsification scheme called ExDyna. In ExDyna, the gradient tensor of the model comprises fined-grained blocks, and contiguous blocks are grouped into non-overlapping partitions. Each worker selects gradients in its exclusively allocated partition so that gradient build-up never occurs. To balance the workload of gradient selection between workers, ExDyna adjusts the topology of partitions by comparing the workloads of adjacent partitions. In addition, ExDyna supports online threshold scaling, which estimates the accurate threshold of gradient selection on-the-fly. Accordingly, ExDyna can satisfy the user-required sparsity level during a training period regardless of models and datasets. Therefore, ExDyna can enhance the scalability of distributed training systems by preserving near-optimal gradient sparsification cost. In experiments, ExDyna outperformed state-of-the-art sparsifiers in terms of training speed and sparsification performance while achieving high accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑