arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.00089cs.LGmath.OCstat.ML
收藏

分期近端优化

Amortized Proximal Optimization

University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(向量人工智能研究所);Stanford University(斯坦福大学)

Juhan Bae, Paul Vicol, Jeff Z. HaoChen, Roger Grosse

摘要:

我们提出了一个在线元优化的参数,管理优化,称为摊销邻近优化(APO)的框架。我们首先解释各种现有的神经网络优化近似随机邻近点的方法,在函数空间和权重空间的邻近项的当前批次的损失。APO背后的想法是通过元学习更新规则的参数来摊销近点目标的最小化。我们展示了如何APO可以用来适应学习率或结构化的预处理矩阵。在适当的假设下,APO可以恢复现有的优化器,如自然梯度下降和KFAC。它具有较低的计算开销,避免了一些二阶优化器所需的昂贵和数值敏感的操作,如矩阵求逆。我们经验性地测试了APO的在线适应学习率和结构化预处理矩阵,用于回归,图像重建,图像分类和自然语言翻译任务。经验上,由APO发现的学习率调度通常优于最佳固定学习率,并且与手动调整的衰减时间表具有竞争力。使用APO来调整结构化预处理矩阵通常会产生与二阶方法相竞争的优化性能。此外,矩阵求逆的不存在提供了数值稳定性,使其对于低精度训练有效。

英文摘要:

We propose a framework for online meta-optimization of parameters that govern optimization, called Amortized Proximal Optimization (APO). We first interpret various existing neural network optimizers as approximate stochastic proximal point methods which trade off the current-batch loss with proximity terms in both function space and weight space. The idea behind APO is to amortize the minimization of the proximal point objective by meta-learning the parameters of an update rule. We show how APO can be used to adapt a learning rate or a structured preconditioning matrix. Under appropriate assumptions, APO can recover existing optimizers such as natural gradient descent and KFAC. It enjoys low computational overhead and avoids expensive and numerically sensitive operations required by some second-order optimizers, such as matrix inverses. We empirically test APO for online adaptation of learning rates and structured preconditioning matrices for regression, image reconstruction, image classification, and natural language translation tasks. Empirically, the learning rate schedules found by APO generally outperform optimal fixed learning rates and are competitive with manually tuned decay schedules. Using APO to adapt a structured preconditioning matrix generally results in optimization performance competitive with second-order methods. Moreover, the absence of matrix inversion provides numerical stability, making it effective for low precision training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑