arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.04902cs.LGcs.CRcs.DCcs.ITmath.IT
收藏

ByzShield:一个高效、鲁棒的分布式训练系统

ByzShield: An Efficient and Robust System for Distributed Training

Iowa State University(爱荷华州立大学)

Konstantinos Konstantinidis, Aditya Ramamoorthy

摘要:

在分布式集群上训练大规模模型是机器学习管道的关键组成部分。然而,如果一些工作者以对抗(拜占庭)方式行事,从而将任意结果返回给参数服务器(PS),则这种训练很容易失败。大量现有的论文考虑了各种攻击模型,并提出了强大的聚合和/或计算冗余,以减轻这些攻击的影响。在这项工作中,我们考虑一个全知的攻击模型,对手有充分的知识梯度计算分配的工人,可以选择攻击(最多)任何q出K个工人节点,以引起最大的损害。我们的基于冗余的方法ByzShield利用二分扩展图的属性将任务分配给工人;这有助于有效地减轻拜占庭行为的影响。具体来说,我们证明了一个上限的最坏情况下的分数的腐败梯度的基础上,我们的建设是基于相互正交的拉丁方和拉马努金图的特征值。我们的数值实验表明,与最先进的技术相比,损坏梯度的比例平均减少了36%以上。同样,我们在CIFAR-10数据集上进行的图像分类训练实验表明,ByzShield在最复杂的攻击下平均具有20%的准确性优势。与以前的工作相比,ByzShield还容忍了更大比例的对抗节点。

英文摘要:

Training of large scale models on distributed clusters is a critical component of the machine learning pipeline. However, this training can easily be made to fail if some workers behave in an adversarial (Byzantine) fashion whereby they return arbitrary results to the parameter server (PS). A plethora of existing papers consider a variety of attack models and propose robust aggregation and/or computational redundancy to alleviate the effects of these attacks. In this work we consider an omniscient attack model where the adversary has full knowledge about the gradient computation assignments of the workers and can choose to attack (up to) any q out of K worker nodes to induce maximal damage. Our redundancy-based method ByzShield leverages the properties of bipartite expander graphs for the assignment of tasks to workers; this helps to effectively mitigate the effect of the Byzantine behavior. Specifically, we demonstrate an upper bound on the worst case fraction of corrupted gradients based on the eigenvalues of our constructions which are based on mutually orthogonal Latin squares and Ramanujan graphs. Our numerical experiments indicate over a 36% reduction on average in the fraction of corrupted gradients compared to the state of the art. Likewise, our experiments on training followed by image classification on the CIFAR-10 dataset show that ByzShield has on average a 20% advantage in accuracy under the most sophisticated attacks. ByzShield also tolerates a much larger fraction of adversarial nodes compared to prior work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑