arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.08461cs.LGstat.ML
收藏

基于超梯度下降的一阶预处理

First-Order Preconditioning via Hypergradient Descent

Uber AI(优步人工智能实验室)

Ted Moskovitz, Rui Wang, Janice Lan, Sanyam Kapoor, Thomas Miconi, Jason Yosinski, Aditya Rawal

摘要:

标准的梯度下降方法容易受到一系列可能阻碍训练的问题的影响,例如高相关性和参数空间中的不同缩放。这些困难可以通过二阶方法来解决,该方法将预处理矩阵应用于梯度以提高收敛性。不幸的是,这种算法通常难以扩展到高维问题,部分原因是计算特定的预处理器,如逆Hessian或Fisher信息矩阵是非常昂贵的。我们引入一阶预处理(FOP),这是一种快速、可扩展的方法,它概括了以前关于超梯度下降的工作(Almeida et al.,1998; Maclaurin等人,2015; Baydin等人,2017)学习只利用一阶信息的预处理矩阵。实验表明,FOP能够以最小的计算开销提高标准深度学习优化器在视觉分类和强化学习任务上的性能。我们还研究了学习的预处理矩阵的性质,并进行了初步的理论分析的算法。

英文摘要:

Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence. Unfortunately, such algorithms typically struggle to scale to high-dimensional problems, in part because the calculation of specific preconditioners such as the inverse Hessian or Fisher information matrix is highly expensive. We introduce first-order preconditioning (FOP), a fast, scalable approach that generalizes previous work on hypergradient descent (Almeida et al., 1998; Maclaurin et al., 2015; Baydin et al.,2017) to learn a preconditioning matrix that only makes use of first-order information. Experiments show that FOP is able to improve the performance of standard deep learning optimizers on visual classification and reinforcement learning tasks with minimal computational overhead. We also investigate the properties of the learned preconditioning matrices and perform a preliminary theoretical analysis of the algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑