使用二阶优化方法训练深度神经网络（DNN）吸引了许多研究人员。最近提出的方法，特征值校正的克罗内克因子分解（EKFAC）（George等人，2018），提出了一种将自然梯度更新视为对角方法的解释，并纠正了Kronecker因子特征基中不准确的重新缩放因子。Gao et al.（2020）考虑了一种新的自然梯度近似，它将Fisher信息矩阵（fisherinformation matrix）近似为一个常数乘以两个矩阵的Kronecker乘积，并保持近似前后的迹相等。在这项工作中，我们结合这两种方法的思想，并提出了迹限制的特征值校正的克罗内克因子分解（TEKFAC）。该方法不仅校正了Kronecker因子化特征基下的不精确重标度因子，还考虑了Gao et al.（2020）提出的新近似方法和有效阻尼技术。我们还讨论了Kronecker因子近似之间的区别和联系。从经验上讲，我们的方法在几个DNN上优于SGD与动量，Adam，EKFAC和TKFAC。
英文摘要：
Using second-order optimization methods for training deep neural networks (DNNs) has attracted many researchers. A recently proposed method, Eigenvalue-corrected Kronecker Factorization (EKFAC) (George et al., 2018), proposes an interpretation of viewing natural gradient update as a diagonal method, and corrects the inaccurate re-scaling factor in the Kronecker-factored eigenbasis. Gao et al. (2020) considers a new approximation to the natural gradient, which approximates the Fisher information matrix (FIM) to a constant multiplied by the Kronecker product of two matrices and keeps the trace equal before and after the approximation. In this work, we combine the ideas of these two methods and propose Trace-restricted Eigenvalue-corrected Kronecker Factorization (TEKFAC). The proposed method not only corrects the inexact re-scaling factor under the Kronecker-factored eigenbasis, but also considers the new approximation method and the effective damping technique proposed in Gao et al. (2020). We also discuss the differences and relationships among the Kronecker-factored approximations. Empirically, our method outperforms SGD with momentum, Adam, EKFAC and TKFAC on several DNNs.
Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks
Centre for Quantum Fields and Gravity（量子场与引力中心）；Department of Physics, Swansea University（斯旺西大学物理系）；School of Mathematical Sciences, Queen Mary University of London（伦敦大学量子玛丽学院数学科学学院）
Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE；Technology Innovation Institute, Abu Dhabi, UAE；Department of Robotics, Khalifa University, Abu Dhabi, UAE
Riemannian Integrated Gradients: A Geometric View of Explainable AI
Center for Theoretical Physics, Polish Academy of Sciences（波兰科学院理论物理中心）；School of Electrical and Mechanical Engineering, The University of Adelaide（阿德莱德大学电气与机械工程学院）
