arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.07706eess.SYcs.SY
收藏

Basil:一种快速、灵活的分散式培训方法

Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training

Ahmed Roushdy Elkordy, Saurav Prakash, A. Salman Avestimehr

摘要:

在去中心化学习环境中检测和缓解拜占庭行为是一项艰巨的任务,特别是当用户的数据分布是异构的时。作为我们的主要贡献,我们提出了Basil,这是一种用于分散式训练系统的快速且计算高效的拜占庭鲁棒算法,它利用了一种新的顺序,内存辅助和基于性能的标准,用于在逻辑环上进行训练,同时过滤拜占庭用户。在IID数据集分布设置中,我们提供了Basil的理论收敛保证,证明了其线性收敛速度。此外,对于IID设置,我们通过实验证明了Basil对各种拜占庭攻击(包括强大的隐藏攻击)具有鲁棒性,同时提供了比最先进的拜占庭弹性分散学习方法更高的测试准确性,高达${\sim}16 \%$。此外,我们通过提出匿名循环数据共享(ACDS)将Basil推广到非IID数据集分布设置,ACDS是一种允许每个节点匿名共享其本地非敏感数据集的随机部分(例如,标志图像)与所有其他节点。我们证明了Basil与ACDS一起,只有$5\%$数据共享提供了有效的容忍拜占庭节点,不像最先进的拜占庭强大的算法,完全失败的异构数据设置。最后,为了减少Basil在逻辑环上顺序实现的整体延迟,我们提出了Basil+。特别是,Basil+通过在逻辑环组之间启用拜占庭健壮的并行训练来提供可扩展性,同时,由于每个组内的顺序训练,它保留了Basil的性能增益。此外,我们通过不同的实验组实验证明了Basil+的可扩展性增益。

英文摘要:

Detection and mitigation of Byzantine behaviors in a decentralized learning setting is a daunting task, especially when the data distribution at the users is heterogeneous. As our main contribution, we propose Basil, a fast and computationally efficient Byzantine robust algorithm for decentralized training systems, which leverages a novel sequential, memory assisted and performance-based criteria for training over a logical ring while filtering the Byzantine users. In the IID dataset distribution setting, we provide the theoretical convergence guarantees of Basil, demonstrating its linear convergence rate. Furthermore, for the IID setting, we experimentally demonstrate that Basil is robust to various Byzantine attacks, including the strong Hidden attack, while providing up to ${\sim}16 \%$ higher test accuracy over the state-of-the-art Byzantine-resilient decentralized learning approach. Additionally, we generalize Basil to the non-IID dataset distribution setting by proposing Anonymous Cyclic Data Sharing (ACDS), a technique that allows each node to anonymously share a random fraction of its local non-sensitive dataset (e.g., landmarks images) with all other nodes. We demonstrate that Basil alongside ACDS with only $5\%$ data sharing provides effective toleration of Byzantine nodes, unlike the state-of-the-art Byzantine robust algorithm that completely fails in the heterogeneous data setting. Finally, to reduce the overall latency of Basil resulting from its sequential implementation over the logical ring, we propose Basil+. In particular, Basil+ provides scalability by enabling Byzantine-robust parallel training across groups of logical rings, and at the same time, it retains the performance gains of Basil due to sequential training within each group. Furthermore, we experimentally demonstrate the scalability gains of Basil+ through different sets of experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑