全新的 K-FACs:通过在线分解更新加速 K-FAC
Brand New K-FACs: Speeding up K-FAC with Online Decomposition Updates
AI总结:
本文利用在线数值线性代数技术提出一种在层大小上呈线性扩展的K-factor逆更新方法,显著降低了K-FAC的计算成本,并在CIFAR10分类任务中比RS-KFAC更快达到目标准确率。
摘要:
K-FAC (arXiv:1503.05671, arXiv:1602.01407) 是深度学习(DL)中自然梯度(NG)的一种可处理实现,其瓶颈在于计算所谓的“Kronecker因子”(K-factors)的逆。RS-KFAC (arXiv:2206.15397) 是一种 K-FAC 的改进方法,它提供了一种低成本的估计 K-factors 逆的方法。在本文中,我们利用 K-factors 的指数平均构造范式,并使用在线数值线性代数技术,提出了一种成本更低(但精度较低)的估计 K-factors 逆的方法。具体而言,我们提出了一种在层大小上呈线性扩展的 K-factor 逆更新。我们还提出了一种同样呈线性扩展的逆应用过程(K-FAC 的过程呈立方级扩展,而 RS-KFAC 的过程呈平方级扩展)。总体而言,我们提出的算法给出了一种近似的 K-FAC 实现,其预处理部分在层大小上呈线性扩展(相比之下,K-FAC 呈立方级扩展,RS-KFAC 呈平方级扩展)。但重要的是,与 RS-KFAC 方法 (arXiv:2206.15397) 不同,这种更新仅在特定情况下适用(通常适用于所有 FC 层)。数值结果表明,通过将我们提出的更新加入 RS-KFAC,可以以最小的 CPU 开销降低 RS-KFAC 的求逆误差。基于所提出的过程、对其的修正以及 RS-KFAC,我们提出了三种用于优化通用深度神经网络的实用算法。数值结果表明,在 CIFAR10 分类任务上使用略微修改版本的 VGG16_bn 时,对于任何目标测试准确率,其中两种算法的性能均优于 RS-KFAC。我们提出的算法在达到 91% 的测试准确率时比 SENG(深度学习经验自然梯度的最先进实现;arXiv:2006.05924)更快,但在更高测试准确率下表现不如 SENG。
英文摘要:
K-FAC (arXiv:1503.05671, arXiv:1602.01407) is a tractable implementation of Natural Gradient (NG) for Deep Learning (DL), whose bottleneck is computing the inverses of the so-called ``Kronecker-Factors'' (K-factors). RS-KFAC (arXiv:2206.15397) is a K-FAC improvement which provides a cheap way of estimating the K-factors inverses. In this paper, we exploit the exponential-average construction paradigm of the K-factors, and use online numerical linear algebra techniques to propose an even cheaper (but less accurate) way of estimating the K-factors inverses. In particular, we propose a K-factor inverse update which scales linearly in layer size. We also propose an inverse application procedure which scales linearly as well (the one of K-FAC scales cubically and the one of RS-KFAC scales quadratically). Overall, our proposed algorithm gives an approximate K-FAC implementation whose preconditioning part scales linearly in layer size (compare to cubic for K-FAC and quadratic for RS-KFAC). Importantly however, this update is only applicable in some circumstances (typically for all FC layers), unlike the RS-KFAC approach (arXiv:2206.15397). Numerical results show RS-KFAC's inversion error can be reduced with minimal CPU overhead by adding our proposed update to it. Based on the proposed procedure, a correction to it, and RS-KFAC, we propose three practical algorithms for optimizing generic Deep Neural Nets. Numerical results show that two of these outperform RS-KFAC for any target test accuracy on CIFAR10 classification with a slightly modified version of VGG16_bn. Our proposed algorithms achieve 91$\%$ test accuracy faster than SENG (the state of art implementation of empirical NG for DL; arXiv:2006.05924) but underperform it for higher test-accuracy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。