arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.03356cs.LG
收藏

M-FAC:二阶信息的有效无矩阵逼近

M-FAC: Efficient Matrix-Free Approximations of Second-Order Information

IST Austria(奥地利科学技术研究所);Neural Magic

Elias Frantar, Eldar Kurtic, Dan Alistarh

摘要:

有效逼近损失函数的局部曲率信息是深度神经网络优化和压缩的关键工具。然而,大多数近似二阶信息的现有方法具有较高的计算或存储成本,这可能限制其实用性。在这项工作中,我们调查矩阵的自由,线性时间的方法来估计逆海森向量积(IHVP)的情况下,海森可以近似为秩一矩阵的总和,在经典的近似海森的经验Fisher矩阵。我们提出了两个新的算法作为一个框架的一部分,称为M-FAC:第一个算法是针对网络压缩,可以计算IHVP的尺寸$d$,如果Hessian是作为一个总和的$m$秩一矩阵,使用$O(dm^2)$预计算,$O(dm)$成本计算IHVP,查询成本$O(m)$的任何单个元素的逆Hessian。第二种算法针对优化设置,其中我们希望计算在优化步骤的滑动窗口上估计的逆Hessian与给定梯度方向之间的乘积,如预处理SGD所需。我们给出了一个算法的成本$O(dm + m^2)$的计算IHVP和$O(dm + m^3)$的添加或删除任何梯度的滑动窗口。这两个算法产生的最先进的结果,网络修剪和优化与较低的计算开销相对于现有的二阶方法。实现可参见[9]和[17]。

英文摘要:

Efficiently approximating local curvature information of the loss function is a key tool for optimization and compression of deep neural networks. Yet, most existing methods to approximate second-order information have high computational or storage costs, which can limit their practicality. In this work, we investigate matrix-free, linear-time approaches for estimating Inverse-Hessian Vector Products (IHVPs) for the case when the Hessian can be approximated as a sum of rank-one matrices, as in the classic approximation of the Hessian by the empirical Fisher matrix. We propose two new algorithms as part of a framework called M-FAC: the first algorithm is tailored towards network compression and can compute the IHVP for dimension $d$, if the Hessian is given as a sum of $m$ rank-one matrices, using $O(dm^2)$ precomputation, $O(dm)$ cost for computing the IHVP, and query cost $O(m)$ for any single element of the inverse Hessian. The second algorithm targets an optimization setting, where we wish to compute the product between the inverse Hessian, estimated over a sliding window of optimization steps, and a given gradient direction, as required for preconditioned SGD. We give an algorithm with cost $O(dm + m^2)$ for computing the IHVP and $O(dm + m^3)$ for adding or removing any gradient from the sliding window. These two algorithms yield state-of-the-art results for network pruning and optimization with lower computational overhead relative to existing second-order methods. Implementations are available at [9] and [17].

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑