arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.13837cs.DC
收藏

通过客户端采样使去中心化学习变得实用

Decentralized Learning Made Practical with Client Sampling

Martijn de Vos, Akash Dhasade, Anne-Marie Kermarrec, Erick Lavoie, Johan Pouwelse, Rishi Sharma

AI总结:

针对去中心化学习中全网参与导致的高通信、慢节点和动态可用性问题,本文提出Plexus,通过去中心化对等采样每轮仅选择少量可用节点训练并聚合模型,显著缩短训练时间、通信量和资源消耗。

摘要:

去中心化学习(DL)利用边缘设备进行协同模型训练,同时避免由中央服务器进行协调。由于隐私方面的顾虑,DL已成为集中式学习方案的一种有吸引力的替代方案,因为训练数据从不离开设备。在一轮DL中,所有节点都参与模型训练,并与其他一些节点交换其模型。在大规模异构网络中执行DL会导致高昂的通信成本,并且由于慢速节点而延长每轮持续时间,实际上增加了总训练时间。此外,当前的DL算法还假设所有节点始终可用于训练和聚合,这降低了DL的实用性。本文提出了Plexus,一个高效、可扩展且实用的DL系统。Plexus:(1)通过引入去中心化对等采样器来避免全网参与,该采样器选择每轮训练模型的可用节点的小子集;(2)每轮聚合由节点生成的已训练模型。Plexus被设计为能够处理节点的加入和离开(节点变动)。我们在实验中纳入边缘设备计算速度、成对延迟、网络容量和可用性的真实轨迹,对Plexus进行了广泛评估。我们在四个常见学习任务上的实验实证表明,与基线DL算法相比,Plexus将达到目标准确率所需时间缩短1.2–8.3倍,通信量减少2.4–15.3倍,收敛所需训练资源减少6.4–370倍。

英文摘要:

Decentralized learning (DL) leverages edge devices for collaborative model training while avoiding coordination by a central server. Due to privacy concerns, DL has become an attractive alternative to centralized learning schemes since training data never leaves the device. In a round of DL, all nodes participate in model training and exchange their model with some other nodes. Performing DL in large-scale heterogeneous networks results in high communication costs and prolonged round durations due to slow nodes, effectively inflating the total training time. Furthermore, current DL algorithms also assume all nodes are available for training and aggregation at all times, diminishing the practicality of DL. This paper presents Plexus, an efficient, scalable, and practical DL system. Plexus (1) avoids network-wide participation by introducing a decentralized peer sampler that selects small subsets of available nodes that train the model each round and, (2) aggregates the trained models produced by nodes every round. Plexus is designed to handle joining and leaving nodes (churn). We extensively evaluate Plexus by incorporating realistic traces for compute speed, pairwise latency, network capacity, and availability of edge devices in our experiments. Our experiments on four common learning tasks empirically show that Plexus reduces time-to-accuracy by 1.2-8.3x, communication volume by 2.4-15.3x and training resources needed for convergence by 6.4-370x compared to baseline DL algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑