Second-order optimization approaches like the generalized Gauss-Newton method are considered more powerful as they utilize the curvature information of the objective function with preconditioning matrices. Albeit offering tempting theoretical benefits, they are not easily applicable to modern deep learning. The major reason is due to the quadratic memory and cubic time complexity to compute the inverse of the matrix. These requirements are infeasible even with state-of-the-art hardware. In this work, we propose Ginger, an eigendecomposition for the inverse of the generalized Gauss-Newton matrix. Our method enjoys efficient linear memory and time complexity for each iteration. Instead of approximating the conditioning matrix, we directly maintain its inverse to make the approximation more accurate. We provide the convergence result of Ginger for non-convex objectives. Our experiments on different tasks with different model architectures verify the effectiveness of our method. Our code is publicly available.
Dynamic Graph Structure Learning via Resistance Curvature Flow
School of Artificial Intelligence, South China Normal University(人工智能学院,华南师范大学);School of Computer Science, South China Normal University(计算机科学学院,华南师范大学);State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(数学科学国家重点实验室,数学与系统科学研究院,中国科学院)
School of Artificial Intelligence, South China Normal University(人工智能学院,华南师范大学);State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(数学科学国家重点实验室,数学与系统科学研究院,中国科学院)
Generalized Quadratic Gradient: A New Direction in Optimization via the Fusion of Positive-Definite Curvature Matrices and Gradients into A Unified Framework
A Self-supervised Mixed-curvature Graph Neural Network
North China Electric Power University(华北电力大学);Beijing University of Posts and Telecommunications(北京邮电大学);Beihang University(北京航空航天大学);University of California, Davis(加州大学戴维斯分校);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)