迈向深度学习中的实用二阶优化器：来自Fisher信息分析的见解
Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis
- Concordia University（康考迪亚大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文提出AdaFisher，一种基于Fisher信息矩阵对角块Kronecker近似的新型自适应二阶优化器，旨在平衡二阶方法的收敛优势与计算效率，在图像分类和语言建模任务中展现出优于Adam和SGD的准确率、收敛速度及稳定性。
AI中文摘要：
一阶优化方法仍是训练深度神经网络（DNN）的标准。像Adam这样的优化器通过用对角矩阵预条件随机梯度来纳入有限的曲率信息。尽管一阶方法被广泛采用，但二阶优化算法通常比Adam和SGD等方法表现出更优的收敛性。然而，由于单次迭代计算成本显著高于一阶方法，它们在训练DNN时的实用性仍受限。本文提出AdaFisher，一种新型自适应二阶优化器，利用Fisher信息矩阵的对角块Kronecker近似来自适应地预条件梯度。AdaFisher旨在弥合二阶方法改善的收敛性与泛化能力与训练DNN所需的计算效率之间的差距。尽管二阶优化器传统上速度较慢，但AdaFisher在图像分类和语言建模等任务中表现出色，在超参数调优期间展现出显著的稳定性和鲁棒性。我们证明AdaFisher在准确率和收敛速度上均优于最先进的优化器。代码可在https://github.com/AtlasAnalyticsLab/AdaFisher获取。
英文摘要：
First-order optimization methods remain the standard for training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by preconditioning the stochastic gradient with a diagonal matrix. Despite the widespread adoption of first-order methods, second-order optimization algorithms often exhibit superior convergence compared to methods like Adam and SGD. However, their practicality in training DNNs is still limited by a significantly higher per-iteration computational cost compared to first-order methods. In this thesis, we present AdaFisher, a novel adaptive second-order optimizer that leverages a diagonal block-Kronecker approximation of the Fisher information matrix to adaptively precondition gradients. AdaFisher aims to bridge the gap between the improved convergence and generalization of second-order methods and the computational efficiency needed for training DNNs. Despite the traditionally slower speed of second-order optimizers, AdaFisher is effective for tasks such as image classification and language modeling, exhibiting remarkable stability and robustness during hyperparameter tuning. We demonstrate that AdaFisher outperforms state-of-the-art optimizers in both accuracy and convergence speed. The code is available from https://github.com/AtlasAnalyticsLab/AdaFisher.
