arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.03295cs.LGcs.AImath.OCstat.ML
收藏

Ginger:一种适用于通用神经网络的线性复杂度高效曲率近似方法

Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks

Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所(Amii));University of Alberta(阿尔伯塔大学);Borealis AI

Yongchang Hao, Yanshuai Cao, Lili Mou

AI总结:

提出Ginger方法,通过特征分解直接维护广义高斯-牛顿矩阵的逆,实现线性复杂度,适用于非凸目标,并在多种任务和架构上验证了有效性。

摘要:

二阶优化方法(如广义高斯-牛顿方法)被认为更强大,因为它们利用目标函数的曲率信息并采用预条件矩阵。尽管这些方法提供了诱人的理论优势,但它们难以应用于现代深度学习。主要原因是计算矩阵逆需要二次内存和三次时间复杂度,即使使用最先进的硬件,这些要求也难以实现。在这项工作中,我们提出了Ginger,一种针对广义高斯-牛顿矩阵逆的特征分解方法。我们的方法在每次迭代中享有高效的线性内存和时间复杂度。我们不近似条件矩阵,而是直接维护其逆矩阵,以使近似更加准确。我们提供了Ginger在非凸目标上的收敛结果。我们在不同任务和不同模型架构上的实验验证了我们方法的有效性。我们的代码已公开。

英文摘要:

Second-order optimization approaches like the generalized Gauss-Newton method are considered more powerful as they utilize the curvature information of the objective function with preconditioning matrices. Albeit offering tempting theoretical benefits, they are not easily applicable to modern deep learning. The major reason is due to the quadratic memory and cubic time complexity to compute the inverse of the matrix. These requirements are infeasible even with state-of-the-art hardware. In this work, we propose Ginger, an eigendecomposition for the inverse of the generalized Gauss-Newton matrix. Our method enjoys efficient linear memory and time complexity for each iteration. Instead of approximating the conditioning matrix, we directly maintain its inverse to make the approximation more accurate. We provide the convergence result of Ginger for non-convex objectives. Our experiments on different tasks with different model architectures verify the effectiveness of our method. Our code is publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑