带加性梯度噪声的一阶方法的速度-鲁棒性权衡
The Speed-Robustness Trade-Off for First-Order Methods with Additive Gradient Noise
AI总结：
本文研究一阶优化方法在收敛速度与对加性梯度噪声敏感性之间的权衡，针对强凸二次函数和光滑强凸函数，提出带标量调节参数的简单解析更新规则，可显式调节速度与鲁棒性，并通过数值实验验证其性能。
摘要：
我们研究一阶优化方法在收敛速度与对随机加性梯度噪声的敏感性之间的权衡。普通梯度下降（GD）可以通过分别增大或减小步长来实现快速但敏感或缓慢但鲁棒。然而，当使用加速方法（如 Polyak 的重球（HB）方法或 Nesterov 的快速梯度（FG）方法）时，如何驾驭这种权衡尚不清楚。我们考虑两类函数：（1）强凸二次函数和（2）光滑强凸函数。对于每一类函数，我们提出一种可计算的方法，用于计算一大类一阶方法的收敛速度和对加性梯度噪声的敏感性，并提出在这些相互竞争的性能指标之间进行权衡的算法设计。每种设计由一个简单的解析更新规则组成，具有两个记忆状态，类似于 HB 和 FG。此外，每种设计都有一个标量调节参数，可以显式地在收敛速度和对加性梯度噪声的敏感性之间进行权衡。我们通过将设计的收敛速度和敏感性与许多其他算法进行比较，并在 Nesterov 的“坏函数”上进行仿真，数值验证了我们设计的性能。
英文摘要：
We study the trade-off between convergence rate and sensitivity to stochastic additive gradient noise for first-order optimization methods. Ordinary Gradient Descent (GD) can be made fast-and-sensitive or slow-and-robust by increasing or decreasing the stepsize, respectively. However, it is not clear how such a trade-off can be navigated when working with accelerated methods such as Polyak's Heavy Ball (HB) or Nesterov's Fast Gradient (FG) methods. We consider two classes of functions: (1) strongly convex quadratics and (2) smooth strongly convex functions. For each function class, we present a tractable way to compute the convergence rate and sensitivity to additive gradient noise for a broad family of first-order methods, and we present algorithm designs that trade off these competing performance metrics. Each design consists of a simple analytic update rule with two states of memory, similar to HB and FG. Moreover, each design has a scalar tuning parameter that explicitly trades off convergence rate and sensitivity to additive gradient noise. We numerically validate the performance of our designs by comparing their convergence rate and sensitivity to those of many other algorithms, and through simulations on Nesterov's "bad function".
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
