arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.06737cs.LGmath.OCstat.ML
收藏

用于深度学习的Kronecker因子化拟牛顿方法

Kronecker-factored Quasi-Newton Methods for Deep Learning

Columbia University(哥伦比亚大学)

Yi Ren, Achraf Bahamou, Donald Goldfarb

摘要:

二阶方法利用比一阶方法更丰富的曲率信息,具有加速优化的能力。然而,大多数对于训练参数数量巨大的深度学习来说都是不切实际的。在Goldfarb等人(2020)中,提出了实用的拟牛顿方法,该方法通过逐层块对角矩阵来近似多层感知器(MLP)模型的Hessian,其中每层的块进一步通过与限制到该层的Hessian结构相对应的Kronecker积来近似。在这里,我们通过分析卷积层的Hessian矩阵的Kronecker因子结构,扩展了这些方法,使它们能够应用于卷积神经网络(CNN)。还提出了对Goldfarb等人(2020)中方法的几项改进,可应用于MLP和CNN。这些新方法的内存要求与一阶方法相当,每次迭代的时间复杂度远低于Goldfarb等人(2020)的方法。此外,收敛结果证明了一个变种在相对温和的条件下。最后,我们将我们的新方法与几种最先进的(SOTA)方法在MLP自动编码器和CNN问题上的性能进行了比较,发现它们优于一阶SOTA方法,并优于二阶SOTA方法。

英文摘要:

Second-order methods have the capability of accelerating optimization by using much richer curvature information than first-order methods. However, most are impractical for deep learning, where the number of training parameters is huge. In Goldfarb et al. (2020), practical quasi-Newton methods were proposed that approximate the Hessian of a multilayer perceptron (MLP) model by a layer-wise block diagonal matrix where each layer's block is further approximated by a Kronecker product corresponding to the structure of the Hessian restricted to that layer. Here, we extend these methods to enable them to be applied to convolutional neural networks (CNNs), by analyzing the Kronecker-factored structure of the Hessian matrix of convolutional layers. Several improvements to the methods in Goldfarb et al. (2020) are also proposed that can be applied to both MLPs and CNNs. These new methods have memory requirements comparable to first-order methods and much less per-iteration time complexity than those in Goldfarb et al. (2020). Moreover, convergence results are proved for a variant under relatively mild conditions. Finally, we compared the performance of our new methods against several state-of-the-art (SOTA) methods on MLP autoencoder and CNN problems, and found that they outperformed the first-order SOTA methods and performed comparably to the second-order SOTA methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑