arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.06993cs.DCcs.NI
收藏

OptiReduce:面向云中分布式深度学习的弹性且尾部最优的AllReduce

OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloud

Ertza Warraich, Omer Shabtai, Khalid Manaa, Shay Vargaftik, Yonatan Piasetzky, Matty Kadosh, Lalith Suresh, Muhammad Shahbaz

AI总结:

提出OptiReduce云集合通信系统,利用DDL弹性处理近似或丢失梯度,通过自适应超时机制与Transpose AllReduce等策略平衡尾部性能与模型精度,在云环境中较Gloo和NCCL分别平均提升70%和30%的TTA。

摘要:

我们提出了OptiReduce,一种面向云的新型集合通信系统,在存在计算(落后节点)和通信(拥塞与梯度丢失)波动时,为深度学习任务提供有界且可预测的完成时间。OptiReduce利用分布式深度学习(DDL)训练与微调的固有弹性及随机性,能够处理近似(或丢失)的梯度——在尾部性能与训练模型精度之间提供高效平衡。基于DDL这一领域特性,OptiReduce引入:(1) 机制(如具有自适应超时的不可靠有界传输)以改善DDL任务的尾部执行时间;(2) 策略(如Transpose AllReduce和Hadamard Transform)以减轻梯度丢失对模型精度的影响。评估表明,在共享云环境(如CloudLab)中,与Gloo和NCCL相比,OptiReduce达到精度的时间(TTA)平均分别快70%和30%。

英文摘要:

We present OptiReduce, a new collective-communication system for the cloud with bounded, predictable completion times for deep-learning jobs in the presence of varying computation (stragglers) and communication (congestion and gradient drops) variabilities. OptiReduce exploits the inherent resiliency and the stochastic nature of distributed deep-learning (DDL) training and fine-tuning to work with approximated (or lost) gradients -- providing an efficient balance between (tail) performance and the resulting accuracy of the trained models. Exploiting this domain-specific characteristic of DDL, OptiReduce introduces (1) mechanisms (e.g., unreliable bounded transport with adaptive timeout) to improve the DDL jobs' tail execution time, and (2) strategies (e.g., Transpose AllReduce and Hadamard Transform) to mitigate the impact of gradient drops on model accuracy. Our evaluation shows that OptiReduce achieves 70% and 30% faster time-to-accuracy (TTA), on average, when operating in shared, cloud environments (e.g., CloudLab) compared to Gloo and NCCL, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑