arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.11352cs.DC
收藏

利用网络感知的自适应树和辅助路由加速地理分布式机器学习

Accelerating Geo-distributed Machine Learning with Network-Aware Adaptive Tree and Auxiliary Route

Zonghang Li, Wenjiao Feng, Weibo Cai, Hongfang Yu, Long Luo, Gang Sun, Hongyang Du, Dusit Niyato

AI总结:

本文提出NetStorm通信调度器,通过多根FAPT拓扑度量、网络感知模块、多路径辅助传输和策略一致性协议,优化广域网参数同步,较MXNET加速6.5~9.2倍。

摘要:

分布式机器学习在地理分布式数据分析中日益普及,促进了分散在不同地区数据中心的数据的协同分析。这种范式消除了将敏感原始数据集中在一个位置的需要,但面临着参数同步延迟高的重大挑战,这源于带宽受限、异构且波动的广域网的约束。先前的研究集中在优化同步拓扑,从星型结构演进到基于树的结构。然而,这些解决方案通常依赖于规则的树结构,并缺乏适当的拓扑度量,导致改进有限。本文提出了NetStorm,这是一种自适应且高效的通信调度器,旨在加速跨地理分布式数据中心的参数同步。首先,它为优化多根FAPT同步拓扑建立了一个有效的度量。其次,开发了一个网络感知模块以获取网络知识,辅助拓扑决策。第三,引入了多路径辅助传输机制以增强网络感知并促进多路径传输。最后,我们设计了策略一致性协议以保证传输策略的无缝更新。实证结果表明,NetStorm显著优于MXNET、MLNET和TSEngine等分布式训练系统,其速度比MXNET快6.5~9.2倍。

英文摘要:

Distributed machine learning is becoming increasingly popular for geo-distributed data analytics, facilitating the collaborative analysis of data scattered across data centers in different regions. This paradigm eliminates the need for centralizing sensitive raw data in one location but faces the significant challenge of high parameter synchronization delays, which stems from the constraints of bandwidth-limited, heterogeneous, and fluctuating wide-area networks. Prior research has focused on optimizing the synchronization topology, evolving from starlike to tree-based structures. However, these solutions typically depend on regular tree structures and lack an adequate topology metric, resulting in limited improvements. This paper proposes NetStorm, an adaptive and highly efficient communication scheduler designed to speed up parameter synchronization across geo-distributed data centers. First, it establishes an effective metric for optimizing a multi-root FAPT synchronization topology. Second, a network awareness module is developed to acquire network knowledge, aiding in topology decisions. Third, a multipath auxiliary transmission mechanism is introduced to enhance network awareness and facilitate multipath transmissions. Lastly, we design policy consistency protocols to guarantee seamless updates of transmission policies. Empirical results demonstrate that NetStorm significantly outperforms distributed training systems like MXNET, MLNET, and TSEngine, with a speedup of 6.5~9.2 times over MXNET.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑