arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2101.00218cs.LGcs.NAmath.NAstat.ML
收藏

一种用于深度学习的迭代K-FAC算法

An iterative K-FAC algorithm for Deep Learning

Yingshi Chen

摘要:

Kronecker因子近似曲率(K-FAC)方法是一种用于深度学习的高效二阶优化算法。它的训练时间比SGD(或其他一阶方法)少,在许多大规模问题中具有相同的精度。K-FAC的关键是将Fisher信息矩阵近似为块对角矩阵,其中每个块是微小Kronecker因子的逆。本文提出了一种新的迭代K-FAC算法--CG-FAC。它采用共轭梯度法来逼近自然梯度。这种CG-FAC方法是无矩阵的,即不需要生成矩阵,也不需要生成Kronecker因子A和G。我们证明了迭代CG-FAC算法的时间复杂度和内存复杂度远小于标准K-FAC算法。

英文摘要:

Kronecker-factored Approximate Curvature (K-FAC) method is a high efficiency second order optimizer for the deep learning. Its training time is less than SGD(or other first-order method) with same accuracy in many large-scale problems. The key of K-FAC is to approximates Fisher information matrix (FIM) as a block-diagonal matrix where each block is an inverse of tiny Kronecker factors. In this short note, we present CG-FAC -- an new iterative K-FAC algorithm. It uses conjugate gradient method to approximate the nature gradient. This CG-FAC method is matrix-free, that is, no need to generate the FIM matrix, also no need to generate the Kronecker factors A and G. We prove that the time and memory complexity of iterative CG-FAC is much less than that of standard K-FAC algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑