标准的梯度下降方法容易受到一系列可能阻碍训练的问题的影响,例如高相关性和参数空间中的不同缩放。这些困难可以通过二阶方法来解决,该方法将预处理矩阵应用于梯度以提高收敛性。不幸的是,这种算法通常难以扩展到高维问题,部分原因是计算特定的预处理器,如逆Hessian或Fisher信息矩阵是非常昂贵的。我们引入一阶预处理(FOP),这是一种快速、可扩展的方法,它概括了以前关于超梯度下降的工作(Almeida et al.,1998; Maclaurin等人,2015; Baydin等人,2017)学习只利用一阶信息的预处理矩阵。实验表明,FOP能够以最小的计算开销提高标准深度学习优化器在视觉分类和强化学习任务上的性能。我们还研究了学习的预处理矩阵的性质,并进行了初步的理论分析的算法。
英文摘要:
Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence. Unfortunately, such algorithms typically struggle to scale to high-dimensional problems, in part because the calculation of specific preconditioners such as the inverse Hessian or Fisher information matrix is highly expensive. We introduce first-order preconditioning (FOP), a fast, scalable approach that generalizes previous work on hypergradient descent (Almeida et al., 1998; Maclaurin et al., 2015; Baydin et al.,2017) to learn a preconditioning matrix that only makes use of first-order information. Experiments show that FOP is able to improve the performance of standard deep learning optimizers on visual classification and reinforcement learning tasks with minimal computational overhead. We also investigate the properties of the learned preconditioning matrices and perform a preliminary theoretical analysis of the algorithm.
Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent
Stanford University(斯坦福大学);ICME, Stanford University(斯坦福大学ICME);Department of Mathematics, Stanford University(斯坦福大学数学系);Department of Management Science and Engineering, Stanford University(斯坦福大学管理科学与工程学院)
An operator preconditioning perspective on training in physics-informed machine learning
ETH Zürich(苏黎世联邦理工学院);Sorbonne Université(索邦大学);Extrality(埃克斯特拉蒂(Extrality));ETH AI Center(苏黎世联邦理工学院人工智能中心);Seminar for Applied Mathematics(应用数学研讨班);Institute of Intelligent Systems and Robotics(智能系统与机器人研究所)
NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation
Department of Statistics University of Georgia(统计学系 佐治亚大学);School of Computing University of Georgia(计算学院 佐治亚大学);School of Electrical Engineering Hanyang University(电气工程学院 洪山大学)
When Does Preconditioning Help or Hurt Generalization?
University of Tokyo(东京大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);Columbia University(哥伦比亚大学);RIKEN Center for Brain Science(理化学研究所脑科学中心);University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(向量人工智能研究所);Google Research(谷歌研究院)
Preconditioning Benefits of Spectral Orthogonalization in Muon
Penn Department of Statistics and Data Science, Wharton School, University of Pennsylvania(宾夕法尼亚大学统计与数据科学系,沃顿商学院,宾夕法尼亚大学);Yale Department of Statistics and Data Science, Yale University(耶鲁大学统计与数据科学系,耶鲁大学);Department of Electrical and Systems Engineering, University of Pennsylvania(宾夕法尼亚大学电气与系统工程系,宾夕法尼亚大学)
SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching
MaLGa;DIBRIS Università degli Studi di Genova(DIBRIS基因瓦大学);Università degli Studi di Genova(基因瓦大学);DIMA Università degli Studi di Genova(DIMA基因瓦大学);IIT Università degli Studi di Genova(IIT基因瓦大学)
Efficient Over-parameterized Matrix Sensing from Noisy Measurements via Alternating Preconditioned Gradient Descent
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences(机器人国家重点实验室,沈阳自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);State University of New York at Buffalo(纽约州立大学布法罗分校);School of Management, Xi’an Jiaotong University(西安交通大学管理学院)
Asymptotic Analysis of Conditioned Stochastic Gradient Descent
École Polytechnique(巴黎综合理工学院);Institut Polytechnique de Paris(巴黎理工学院);ENSAI(法国国立统计与信息分析学院);École Nationale de la Statistique et de l’Analyse de l’Information(法国国立统计与信息分析学院)
Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra
University of Pavia(帕维亚大学);Department of Mathematics(数学系);Department of Civil Engineering and Architecture(土木工程与建筑系);Institut Camille Jordan, Lyon 1 Université(让·乔丹研究所,里昂1大学)
Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
Department of Computer Science University of British Columbia(计算机科学系不列颠哥伦比亚大学);University of British Columbia Canada(不列颠哥伦比亚大学加拿大);CIFAR AI Chair (Amii) &(CIFAR人工智能主席(Amii));Department of Electrical and Computer Engineering University of British Columbia(电气与计算机工程系不列颠哥伦比亚大学)
Bilevel Learning via Strategically Inexact Stochastic Gradient Descent
EPSRC Centre for Doctoral Training in Statistical Applied Mathematics at Bath(巴斯大学统计应用数学博士培训中心);Australian Research Council(澳大利亚研究理事会);Indian Institute of Technology (IIT), Kharagpur(印度理工学院(Kharagpur分校));University of Melbourne(墨尔本大学);Department of Electronics & Electrical Communication Engineering(电子与电气通信工程系);School of Mathematics and Statistics(数学与统计学学院)
How do simple rotations affect the implicit bias of Adam?
Committee on Computational and Applied Mathematics, University of Chicago(计算与应用数学委员会,芝加哥大学);Department of Electrical & Computer Engineering, University of Washington(电气与计算机工程系,华盛顿大学);NSF-Simons National Institute for Theory and Mathematics in Biology(NSF-西蒙斯国家理论与生物数学研究所);Department of Statistics, Department of Computer Science, and Data Science Institute, University of Chicago(统计系、计算机科学系和数据科学研究所,芝加哥大学)