Overshoot:在基于动量的随机优化中利用未来梯度
Overshoot: Taking advantage of future gradients in momentum-based stochastic optimization
AI总结:
提出Overshoot优化方法,通过在沿动量方向平移后的权重处计算梯度来利用未来梯度,可融入SGD与Adam并实现至少15%的平均步数节省。
摘要:
Overshoot是一种新颖的、基于动量的随机梯度下降优化方法,旨在将性能提升至标准动量和Nesterov动量之上。在传统动量方法中,来自先前步骤的梯度会在执行一步并更新模型之前,与当前模型权重处的梯度进行聚合。Overshoot并不在当前模型权重处计算梯度,而是在沿当前动量方向平移后的模型权重处计算梯度。这牺牲了使用相对于当前确切模型权重的梯度所带来的即时收益,转而在一个很可能与未来更新更相关的点上进行评估。我们表明,将这一原则融入基于动量的优化器(带动量的SGD和Adam)会带来更快的收敛(平均至少节省15%的步骤)。在广泛的任务中,Overshoot始终优于标准动量和Nesterov动量,并且能够集成到流行的基于动量的优化器中,且具有零内存开销和很小的计算开销。
英文摘要:
Overshoot is a novel, momentum-based stochastic gradient descent optimization method designed to enhance performance beyond standard and Nesterov's momentum. In conventional momentum methods, gradients from previous steps are aggregated with the gradient at current model weights before taking a step and updating the model. Rather than calculating gradient at the current model weights, Overshoot calculates the gradient at model weights shifted in the direction of the current momentum. This sacrifices the immediate benefit of using the gradient w.r.t. the exact model weights now, in favor of evaluating at a point, which will likely be more relevant for future updates. We show that incorporating this principle into momentum-based optimizers (SGD with momentum and Adam) results in faster convergence (saving on average at least 15% of steps). Overshoot consistently outperforms both standard and Nesterov's momentum across a wide range of tasks and integrates into popular momentum-based optimizers with zero memory and small computational overhead.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。