基于Kronecker乘积奇异值分解的神经网络中Fisher矩阵的有效逼近
Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition
摘要：
几项研究表明，自然梯度下降比普通的基于梯度下降的方法更有效地最小化目标函数的能力。然而，这种用于训练深度神经网络的方法的瓶颈在于在每次迭代时求解对应于Fisher信息矩阵（Fisher Information Matrix，NIM）的大型密集线性系统的成本过高。这激发了各种近似的精确的或经验的。其中最复杂的是KFAC，它涉及到一个Kronecker分解的块对角近似。仅需少量的额外成本，从精度的角度提出了KFAC的一些改进。这四种新方法的共同特点是它们依赖于一个直接的最小化问题，其解可以通过克罗内克积奇异值分解技术计算。在三个标准深度自动编码器基准上的实验结果表明，它们提供了更准确的近似值。此外，它们在优化速度方面优于KFAC和最先进的一阶方法。
英文摘要：
Several studies have shown the ability of natural gradient descent to minimize the objective function more efficiently than ordinary gradient descent based methods. However, the bottleneck of this approach for training deep neural networks lies in the prohibitive cost of solving a large dense linear system corresponding to the Fisher Information Matrix (FIM) at each iteration. This has motivated various approximations of either the exact FIM or the empirical one. The most sophisticated of these is KFAC, which involves a Kronecker-factored block diagonal approximation of the FIM. With only a slight additional cost, a few improvements of KFAC from the standpoint of accuracy are proposed. The common feature of the four novel methods is that they rely on a direct minimization problem, the solution of which can be computed via the Kronecker product singular value decomposition technique. Experimental results on the three standard deep auto-encoder benchmarks showed that they provide more accurate approximations to the FIM. Furthermore, they outperform KFAC and state-of-the-art first-order methods in terms of optimization speed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
