Second-order methods have the capability of accelerating optimization by using much richer curvature information than first-order methods. However, most are impractical for deep learning, where the number of training parameters is huge. In Goldfarb et al. (2020), practical quasi-Newton methods were proposed that approximate the Hessian of a multilayer perceptron (MLP) model by a layer-wise block diagonal matrix where each layer's block is further approximated by a Kronecker product corresponding to the structure of the Hessian restricted to that layer. Here, we extend these methods to enable them to be applied to convolutional neural networks (CNNs), by analyzing the Kronecker-factored structure of the Hessian matrix of convolutional layers. Several improvements to the methods in Goldfarb et al. (2020) are also proposed that can be applied to both MLPs and CNNs. These new methods have memory requirements comparable to first-order methods and much less per-iteration time complexity than those in Goldfarb et al. (2020). Moreover, convergence results are proved for a variant under relatively mild conditions. Finally, we compared the performance of our new methods against several state-of-the-art (SOTA) methods on MLP autoencoder and CNN problems, and found that they outperformed the first-order SOTA methods and performed comparably to the second-order SOTA methods.
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全国家重点实验室,计算技术研究所)
Eigenvalue-corrected Natural Gradient Based on a New Approximation
Tianjin University(天津大学);Huawei Technologies(华为技术有限公司);Beijing University of Technology(北京工业大学)
使用二阶优化方法训练深度神经网络(DNN)吸引了许多研究人员。最近提出的方法,特征值校正的克罗内克因子分解(EKFAC)(George等人,2018),提出了一种将自然梯度更新视为对角方法的解释,并纠正了Kronecker因子特征基中不准确的重新缩放因子。Gao et al.(2020)考虑了一种新的自然梯度近似,它将Fisher信息矩阵(fisherinformation matrix)近似为一个常数乘以两个矩阵的Kronecker乘积,并保持近似前后的迹相等。在这项工作中,我们结合这两种方法的思想,并提出了迹限制的特征值校正的克罗内克因子分解(TEKFAC)。该方法不仅校正了Kronecker因子化特征基下的不精确重标度因子,还考虑了Gao et al.(2020)提出的新近似方法和有效阻尼技术。我们还讨论了Kronecker因子近似之间的区别和联系。从经验上讲,我们的方法在几个DNN上优于SGD与动量,Adam,EKFAC和TKFAC。
Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
School of Computing, Tokyo Institute of Technology(东京工业大学计算学院);NVIDIA(英伟达);Global Scientific Information and Computing Center, Tokyo Institute of Technology(东京工业大学全球科学信息与计算中心);RIKEN Center for Computational Science(理化学研究所计算科学中心)
Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures
University of Cambridge(剑桥大学);ETH Zurich(苏黎世联邦理工学院);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Tübingen(图宾根大学);Tübingen AI Center(图宾根人工智能中心)
KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
University of Chicago(芝加哥大学);University of Texas at Austin(得克萨斯大学奥斯汀分校);Texas Advanced Computing Center(得克萨斯高级计算中心);University of Wisconsin, Madison(威斯康星大学麦迪逊分校);Argonne National Laboratory(阿贡国家实验室)