具有近最优L2增益和$H_\infty$性能的鲁棒稳定加速动量方法
Robustly Stable Accelerated Momentum Methods With A Near-Optimal L2 Gain and $H_\infty$ Performance
AI总结:
研究了一阶通用动量方法(GMM)在梯度误差下的收敛与鲁棒性权衡,利用$H_\infty$范数和诱导$\ell_2$增益分析其鲁棒稳定性,证明HB虽快但不如NAG鲁棒。
摘要:
我们考虑最小化强凸光滑函数的问题,其中梯度受到平方可加的最坏情况确定性误差的影响。我们研究了在设计一阶算法参数时,收敛速度与对梯度误差的鲁棒性之间的权衡。我们关注一类具有恒定步长和动量参数的通用动量方法(GMM),它可以恢复梯度下降、Nesterov加速梯度、重球和三重动量方法作为特例。我们通过迭代过程中的累积次优性除以梯度误差的$\ell_2$范数来衡量算法的鲁棒性,这可以解释为代表GMM迭代的变换动力系统的最小(诱导)$\ell_2$增益,其中输入是梯度误差序列,输出是到最优点的加权距离。对于二次目标函数,我们利用其与GMM对应的动力系统的$H_\infty$范数的联系显式地计算了诱导$\ell_2$增益,并通过闭式公式构造了最坏情况梯度误差序列。我们还通过刻画GMM系统的结构实稳定半径及其与$H_\infty$范数的联系,研究了GMM在各种设置下对乘性噪声的稳定性。这使我们能够在鲁棒性方面比较GD、HB、NAG方法,并论证HB尽管速度最快,但不如NAG鲁棒……
英文摘要:
We consider the problem of minimizing a strongly convex smooth function where the gradients are subject to additive worst-case deterministic errors that are square-summable. We study the trade-offs between the convergence rate and robustness to gradient errors when designing the parameters of a first-order algorithm. We focus on a general class of momentum methods (GMM) with constant stepsize and momentum parameters which can recover gradient descent, Nesterov's accelerated gradient, the heavy-ball and the triple momentum methods as special cases. We measure the robustness of an algorithm in terms of the cumulative suboptimality over the iterations divided by the $\ell_2$ norm of the gradient errors, which can be interpreted as the minimal (induced) $\ell_2$ gain of a transformed dynamical system that represents the GMM iterations where the input is the gradient error sequence and the output is a weighted distance to the optimum. For quadratic objectives, we compute the induced $\ell_2$ gain explicitly leveraging its connections to the $H_\infty$ norm of the dynamical system corresponding to GMM and construct worst-case gradient error sequences by a closed-form formula. We also study the stability of GMM with respect to multiplicative noise in various settings by characterizing the structured real and stability radius of the GMM system through their connections to the $H_\infty$ norm. This allows us to compare GD, HB, NAG methods in terms of robustness, and argue that HB is not as robust as NAG despite being the fastest...
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。