arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.00489cs.LGmath.OC
收藏

高效基于符号的优化:通过方差缩减加速收敛

Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction

Nanjing University(南京大学);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Pazhou Laboratory (Huangpu)(琶洲实验室(黄埔))

Wei Jiang, Sifan Yang, Wenhao Yang, Lijun Zhang

AI总结:

针对signSGD收敛速率问题,提出基于符号的随机方差缩减方法,提升了单机及分布式场景下的收敛速率,实验验证了其有效性。

摘要:

符号随机梯度下降(signSGD)是一种通信高效的方法,仅传输随机梯度的符号用于参数更新。现有研究已证明signSGD可达到$\u27e8d^{1/2}T^{-1/4}\u27e9$的收敛速率,其中$d$表示维度,$T$为迭代次数。本文提出基于符号的随机方差缩减(SSVR)方法,使用方差缩减估计器追踪梯度并利用其符号进行更新,将收敛速率提升至$\u27e8d^{1/2}T^{-1/3}\u27e9$。针对有限和问题,我们的方法可进一步优化,达到$\u27e8m^{1/4}d^{1/2}T^{-1/2}\u27e9$的收敛速率,其中$m$表示分量函数的数量。此外,我们研究了分布式场景下的异构多数投票机制,提出两种新算法,分别实现$\u27e8d^{1/2}T^{-1/2} + dn^{-1/2}\u27e9$和$\u27e8d^{1/4}T^{-1/4}\u27e9$的改进收敛速率,优于此前$\u27e8dT^{-1/4} + dn^{-1/2}\u27e9$和$\u27e8d^{3/8}T^{-1/8}\u27e9$的结果,其中$n$表示节点数。不同任务上的数值实验验证了所提方法的有效性。

英文摘要:

Sign stochastic gradient descent (signSGD) is a communication-efficient method that transmits only the sign of stochastic gradients for parameter updating. Existing literature has demonstrated that signSGD can achieve a convergence rate of $\mathcal{O}(d^{1/2}T^{-1/4})$, where $d$ represents the dimension and $T$ is the iteration number. In this paper, we improve this convergence rate to $\mathcal{O}(d^{1/2}T^{-1/3})$ by introducing the Sign-based Stochastic Variance Reduction (SSVR) method, which employs variance reduction estimators to track gradients and leverages their signs to update. For finite-sum problems, our method can be further enhanced to achieve a convergence rate of $\mathcal{O}(m^{1/4}d^{1/2}T^{-1/2})$, where $m$ denotes the number of component functions. Furthermore, we investigate the heterogeneous majority vote in distributed settings and introduce two novel algorithms that attain improved convergence rates of $\mathcal{O}(d^{1/2}T^{-1/2} + dn^{-1/2})$ and $\mathcal{O}(d^{1/4}T^{-1/4})$ respectively, outperforming the previous results of $\mathcal{O}(dT^{-1/4} + dn^{-1/2})$ and $\mathcal{O}(d^{3/8}T^{-1/8})$, where $n$ represents the number of nodes. Numerical experiments across different tasks validate the effectiveness of our proposed methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑