混合固定:异质结下最优拜占庭ML的配方
Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneity
摘要:
拜占庭机器学习(ML)旨在确保分布式学习算法对行为不端(或拜占庭)机器的弹性。虽然这个问题得到了极大的关注,以前的作品往往假设由机器所持有的数据是同质的,这在实际设置中很少是真的。数据异构性使拜占庭ML更具挑战性,因为拜占庭机器几乎无法与非拜占庭离群值区分开来。已经提出了一些解决方案来解决这个问题,但这些方案提供了次优的概率保证,并且在实践中效果不佳。本文缩小了理论差距,实现了最优性,并得出了良好的实证结果。事实上,我们展示了如何通过一个强大的机制,我们称之为最近邻混合(NNM),自动调整(同质)拜占庭ML的现有解决方案,以适应异构环境,该机制增强了任何标准的鲁棒分布式梯度下降变体,以在异构环境下产生最佳的拜占庭弹性。我们获得了类似的保证(在预期中)通过插入NNM的分布式随机重球方法,一个实用的替代分布式梯度下降。我们获得的经验结果显着优于最先进的拜占庭ML解决方案。
英文摘要:
Byzantine machine learning (ML) aims to ensure the resilience of distributed learning algorithms to misbehaving (or Byzantine) machines. Although this problem received significant attention, prior works often assume the data held by the machines to be homogeneous, which is seldom true in practical settings. Data heterogeneity makes Byzantine ML considerably more challenging, since a Byzantine machine can hardly be distinguished from a non-Byzantine outlier. A few solutions have been proposed to tackle this issue, but these provide suboptimal probabilistic guarantees and fare poorly in practice. This paper closes the theoretical gap, achieving optimality and inducing good empirical results. In fact, we show how to automatically adapt existing solutions for (homogeneous) Byzantine ML to the heterogeneous setting through a powerful mechanism, we call nearest neighbor mixing (NNM), which boosts any standard robust distributed gradient descent variant to yield optimal Byzantine resilience under heterogeneity. We obtain similar guarantees (in expectation) by plugging NNM in the distributed stochastic heavy ball method, a practical substitute to distributed gradient descent. We obtain empirical results that significantly outperform state-of-the-art Byzantine ML solutions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。