arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.16248cs.LG
收藏

Accordion:通过临界学习机制识别的自适应梯度通信

Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification

University of Wisconsin-Madison(威斯康星大学麦迪逊分校)

Saurabh Agarwal, Hongyi Wang, Kangwook Lee, Shivaram Venkataraman, Dimitris Papailiopoulos

摘要:

分布式模型训练由于频繁的模型更新在计算节点之间传输而遭受通信瓶颈。为了缓解这些瓶颈,从业者使用梯度压缩技术,如稀疏化,量化或低秩更新。这些技术通常需要选择静态压缩比,通常需要用户在模型精度和每次迭代加速之间进行权衡。在这项工作中,我们表明,这种性能下降,由于选择一个高的压缩比是不是根本的。自适应压缩策略可以减少通信,同时保持最终的测试精度。受最近关于临界学习机制的研究结果的启发,其中小的梯度误差可能对模型性能产生不可恢复的影响,我们提出了一种简单而有效的自适应压缩算法Accordion。虽然Accordion平均保持了足够高的压缩率,但它避免了在关键学习机制中过度压缩梯度,这是通过一个简单的基于梯度范数的标准来检测的。我们对分布式环境中的许多机器学习任务进行了广泛的实验研究,结果表明,Accordion保持了与未压缩训练相似的模型精度,但与静态方法相比,压缩效果提高了5.5倍,端到端加速提高了4.1倍。我们表明,手风琴也适用于调整批量大小,另一种流行的策略,以缓解通信瓶颈。

英文摘要:

Distributed model training suffers from communication bottlenecks due to frequent model updates transmitted across compute nodes. To alleviate these bottlenecks, practitioners use gradient compression techniques like sparsification, quantization, or low-rank updates. The techniques usually require choosing a static compression ratio, often requiring users to balance the trade-off between model accuracy and per-iteration speedup. In this work, we show that such performance degradation due to choosing a high compression ratio is not fundamental. An adaptive compression strategy can reduce communication while maintaining final test accuracy. Inspired by recent findings on critical learning regimes, in which small gradient errors can have irrecoverable impact on model performance, we propose Accordion a simple yet effective adaptive compression algorithm. While Accordion maintains a high enough compression rate on average, it avoids over-compressing gradients whenever in critical learning regimes, detected by a simple gradient-norm based criterion. Our extensive experimental study over a number of machine learning tasks in distributed environments indicates that Accordion, maintains similar model accuracy to uncompressed training, yet achieves up to 5.5x better compression and up to 4.1x end-to-end speedup over static approaches. We show that Accordion also works for adjusting the batch size, another popular strategy for alleviating communication bottlenecks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑