简化基于动量的正定子流形优化及其在深度学习中的应用
Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
AI总结：
本文提出一种广义黎曼法坐标方法，将带动量的正定子流形优化问题转化为欧几里得无约束问题，并据此开发了仅用矩阵乘法的无逆二阶优化器，适用于低精度深度学习。
摘要：
在黎曼子流形上使用动量进行优化在计算上具有挑战性，因为为了确保迭代点保持在子流形上，我们通常需要求解困难的微分方程。在此，我们针对一类具有仿射不变度量的稀疏或结构化对称正定矩阵简化了此类困难。为此，我们提出了黎曼法坐标的广义版本，该版本动态地正交化度量，并将问题局部转化为欧几里得空间中的无约束问题。我们利用该方法简化了现有用于结构化协方差的方法，并仅通过矩阵乘法开发了适用于低精度深度学习的无矩阵逆的二阶优化器。代码：https://github.com/yorkerlin/StructuredNGD-DL
英文摘要：
Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of sparse or structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. We use our approach to simplify existing approaches for structured covariances and develop matrix-inverse-free $2^\text{nd}$-order optimizers for deep learning with low precision by using only matrix multiplications. Code: https://github.com/yorkerlin/StructuredNGD-DL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
