arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.08161cs.LGstat.ML
收藏

分布式非独立同分布下的拜占庭鲁棒方差缩减联邦学习数据

Byzantine-Robust Variance-Reduced Federated Learning over Distributed Non-i.i.d. Data

Sun Yat-Sen University(中山大学);Pazhou Lab(琶洲实验室);Rensselaer Polytechnic Institute(伦斯勒理工学院)

Jie Peng, Zhaoxian Wu, Qing Ling, Tianyi Chen

摘要:

我们考虑联邦学习问题,其中工人的数据不是独立同分布的(i.i.d.)。在学习过程中,未知数量的Byzantine工人可能会向中心节点发送恶意消息,导致显著的学习错误。大多数拜占庭健壮方法通过使用健壮的聚合规则来聚合接收到的消息来解决这个问题,但是依赖于所有常规工作者都具有i.i.d.的假设。数据,这在许多联邦学习应用程序中并非如此。鉴于减少随机梯度噪声对减轻拜占庭攻击的影响的重要性,我们使用了一种恢复策略来减少内部变化(描述每个常规工人上的样本异质性)和外部变化(描述常规工人之间的样本异质性)的影响,以及随机平均梯度算法来逐渐消除内部变化。然后用一个鲁棒的几何中值算子聚合方差减小的消息。我们证明了所提出的方法达到最优解的一个邻域的线性收敛速度和学习误差是由拜占庭工人的数量。数值实验证实了理论结果,并表明该方法优于国家的最先进的非独立同分布。设置.

英文摘要:

We consider the federated learning problem where data on workers are not independent and identically distributed (i.i.d.). During the learning process, an unknown number of Byzantine workers may send malicious messages to the central node, leading to remarkable learning error. Most of the Byzantine-robust methods address this issue by using robust aggregation rules to aggregate the received messages, but rely on the assumption that all the regular workers have i.i.d. data, which is not the case in many federated learning applications. In light of the significance of reducing stochastic gradient noise for mitigating the effect of Byzantine attacks, we use a resampling strategy to reduce the impact of both inner variation (that describes the sample heterogeneity on every regular worker) and outer variation (that describes the sample heterogeneity among the regular workers), along with a stochastic average gradient algorithm to gradually eliminate the inner variation. The variance-reduced messages are then aggregated with a robust geometric median operator. We prove that the proposed method reaches a neighborhood of the optimal solution at a linear convergence rate and the learning error is determined by the number of Byzantine workers. Numerical experiments corroborate the theoretical results and show that the proposed method outperforms the state-of-the-arts in the non-i.i.d. setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑