arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.12677math.OCcs.LGstat.ML
收藏

一般方差条件下的分布式随机优化

Distributed Stochastic Optimization under a General Variance Condition

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Kun Huang, Xiao Li, Shi Pu

AI总结:

该研究针对分布式随机优化算法理论依赖强梯度有界条件、数据异质性刻画难的问题,在温和方差条件下证明了FedAvg和SCAFFOLD的收敛性,并提出更具信息量的数据异质性度量方式。

摘要:

分布式随机优化因在求解大规模机器学习问题中的有效性,近年来受到广泛关注。尽管已有大量算法被提出并成功应用于各类实际问题,但它们的理论保证主要依赖于随机梯度的某些有界性条件,从一致有界性到放宽的增长条件不等。此外,如何刻画智能体间的数据异质性及其对算法性能的影响仍是一个挑战。基于上述动机,我们重新研究了用于求解分布式随机优化问题的经典Federated Averaging(FedAvg,联邦平均)算法(McMahan等人,2017)以及较新的SCAFFOLD方法(Karimireddy等人,2020),并在光滑非凸目标函数的随机梯度仅满足温和方差条件的情况下,建立了收敛性结果。在该条件下,还证明了算法几乎必然收敛到平稳点。此外,我们讨论了一种更具信息量的数据异质性度量方式及其相关影响。

英文摘要:

Distributed stochastic optimization has drawn great attention recently due to its effectiveness in solving large-scale machine learning problems. Though numerous algorithms have been proposed and successfully applied to general practical problems, their theoretical guarantees mainly rely on certain boundedness conditions on the stochastic gradients, varying from uniform boundedness to the relaxed growth condition. In addition, how to characterize the data heterogeneity among the agents and its impacts on the algorithmic performance remains challenging. In light of such motivations, we revisit the classical Federated Averaging (FedAvg) algorithm (McMahan et al., 2017) as well as the more recent SCAFFOLD method (Karimireddy et al., 2020) for solving the distributed stochastic optimization problem and establish the convergence results under only a mild variance condition on the stochastic gradients for smooth nonconvex objective functions. Almost sure convergence to a stationary point is also established under the condition. Moreover, we discuss a more informative measurement for data heterogeneity as well as its implications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑