arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.06835cs.LGmath.OCstat.ML
收藏

自适应梯度方法通过过参数化收敛得更快(但您应该进行线搜索)

Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)

University of Alberta(阿尔伯塔大学);Mila, McGill University(米拉研究所,麦吉尔大学);University of British Columbia(不列颠哥伦比亚大学);Cornell University(康奈尔大学);Mila, Université de Montréal(米拉研究所,蒙特利尔大学)

Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, Simon Lacoste-Julien

摘要:

自适应梯度方法通常用于训练过参数化模型。为了更好地理解它们的行为,我们研究了一个简单的设置-平滑,凸损失与模型过度参数化足以插值数据。在这种情况下,我们证明了AMSGrad与恒定的步长和动量收敛到极小值在一个更快的$O(1/T)$率。当插值仅近似满足时,恒定步长AMSGrad以相同的速率收敛到解的邻域,而AdaGrad对插值的违反具有鲁棒性。然而,即使对于满足插值的简单凸问题,这两种方法的经验性能在很大程度上取决于步长,需要调整,质疑其自适应性。我们通过使用随机线搜索或Polyak步长自动确定步长来缓解这个问题。通过这些技术,我们证明了AdaGrad和AMSGrad都保留了它们的收敛保证,而不需要知道问题相关的常数。从经验上讲,我们证明了这些技术可以提高自适应梯度方法在不同任务中的收敛性和泛化能力,从使用内核映射的二进制分类到使用深度网络的多类分类。

英文摘要:

Adaptive gradient methods are typically used for training over-parameterized models. To better understand their behaviour, we study a simplistic setting -- smooth, convex losses with models over-parameterized enough to interpolate the data. In this setting, we prove that AMSGrad with constant step-size and momentum converges to the minimizer at a faster $O(1/T)$ rate. When interpolation is only approximately satisfied, constant step-size AMSGrad converges to a neighbourhood of the solution at the same rate, while AdaGrad is robust to the violation of interpolation. However, even for simple convex problems satisfying interpolation, the empirical performance of both methods heavily depends on the step-size and requires tuning, questioning their adaptivity. We alleviate this problem by automatically determining the step-size using stochastic line-search or Polyak step-sizes. With these techniques, we prove that both AdaGrad and AMSGrad retain their convergence guarantees, without needing to know problem-dependent constants. Empirically, we demonstrate that these techniques improve the convergence and generalization of adaptive gradient methods across tasks, from binary classification with kernel mappings to multi-class classification with deep networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑