arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1806.02508cs.DCcs.AIcs.LGcs.PF
收藏

Semi-Dynamic Load Balancing:非专用环境下的高效分布式学习

Semi-Dynamic Load Balancing: Efficient Distributed Learning in Non-Dedicated Environments

HKUST(香港科技大学);University of Toronto(多伦多大学)

Chen Chen, Qizhen Weng, Wei Wang, Baochun Li, Bo Li

摘要:

机器学习(ML)模型越来越多地在集群中训练,其中非专用工作者拥有异构资源。在这种情况下,模型训练效率可能会受到落后者的负面影响-这些工人比其他人跑得慢得多。有效的模型训练需要消除这些落后者,但对于现代ML工作负载,现有的负载均衡策略效率低下,甚至不可行。在本文中,我们提出了一种称为半动态负载均衡的新策略,以消除分布式ML工作负载的落伍者。关键的见解是ML工作者应该在迭代边界上进行负载平衡,对迭代内执行无干扰。我们开发的LB-BSP基于这样的见解,这是一个集成的工人协调机制,适应工人的负载,以他们的瞬时处理能力,通过正确的大小的样本批次在同步障碍。针对CPU和GPU集群的特点,分别为它们定制了批量大小算法。LB-BSP已被实现为TensorFlow和PyTorch等ML框架的Python模块。我们的EC2部署证实了LB-BSP是实用的,有效的和轻量级的,并且能够加速高达$54\%$的分布式训练。

英文摘要:

Machine learning (ML) models are increasingly trained in clusters with non-dedicated workers possessing heterogeneous resources. In such scenarios, model training efficiency can be negatively affected by stragglers -- workers that run much slower than others. Efficient model training requires eliminating such stragglers, yet for modern ML workloads, existing load balancing strategies are inefficient and even infeasible. In this paper, we propose a novel strategy called semi-dynamic load balancing to eliminate stragglers of distributed ML workloads. The key insight is that ML workers shall be load-balanced at iteration boundaries, being non-intrusive to intra-iteration execution. We develop LB-BSP based on such an insight, which is an integrated worker coordination mechanism that adapts workers' load to their instantaneous processing capabilities by right-sizing the sample batches at the synchronization barriers. We have custom-designed the batch sizing algorithm respectively for CPU and GPU clusters based on their own characteristics. LB-BSP has been implemented as a Python module for ML frameworks like TensorFlow and PyTorch. Our EC2 deployment confirms that LB-BSP is practical, effective and light-weight, and is able to accelerating distributed training by up to $54\%$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑