arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1906.02702math.OCcs.DCcs.LGcs.MA
收藏

分布式随机梯度下降系统过渡时间的精确估计

A Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent

Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Boston University(波士顿大学)

Shi Pu, Alex Olshevsky, Ioannis Ch. Paschalidis

摘要:

本文研究了在网络环境下,各代理之间可以相互通信和交换信息时,最小化$n$费用函数的平均值问题。我们考虑只有噪声梯度信息可用的设置。为了解决这个问题,我们研究了分布式随机梯度下降(DSGD)方法,并进行了非渐近收敛分析。对于强凸和光滑的目标函数,DSGD渐近实现最佳的网络独立的收敛速度相比,集中式随机梯度下降(SGD)。我们的主要贡献是表征DSGD接近渐近收敛速度所需的瞬态时间,我们表现为$K_T=\mathcal{O}\left(\frac{n}{(1-ρ_w)^2}\right)$,其中$1-ρ_w$表示混合矩阵的谱隙。此外,我们构造了一个“硬”的最优化问题,我们表明所需的过渡时间DSGD接近渐近收敛速度是下有界的$Ω\left(\frac{n}{(1-ρ_w)^2} \right)$,这意味着所获得的结果的锋利。数值实验证明了理论结果的严密性。

英文摘要:

This paper is concerned with minimizing the average of $n$ cost functions over a network in which agents may communicate and exchange information with each other. We consider the setting where only noisy gradient information is available. To solve the problem, we study the distributed stochastic gradient descent (DSGD) method and perform a non-asymptotic convergence analysis. For strongly convex and smooth objective functions, DSGD asymptotically achieves the optimal network independent convergence rate compared to centralized stochastic gradient descent (SGD). Our main contribution is to characterize the transient time needed for DSGD to approach the asymptotic convergence rate, which we show behaves as $K_T=\mathcal{O}\left(\frac{n}{(1-ρ_w)^2}\right)$, where $1-ρ_w$ denotes the spectral gap of the mixing matrix. Moreover, we construct a "hard" optimization problem for which we show the transient time needed for DSGD to approach the asymptotic convergence rate is lower bounded by $Ω\left(\frac{n}{(1-ρ_w)^2} \right)$, implying the sharpness of the obtained result. Numerical experiments demonstrate the tightness of the theoretical results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑