我们能去掉自适应梯度方法中的平方根吗？一个二阶视角
Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective
AI总结：
本研究从二阶视角探究去掉自适应梯度方法中的平方根，发现无平方根方法在卷积架构上缩小与SGD的泛化差距且在transformer上性能不变，还为半精度非对角方法开发提供了新思路。
摘要：
像Adam(W)这样的自适应梯度优化器是许多深度学习架构（如transformer）的默认训练算法。它们的对角预处理器基于梯度外积，该外积通过平方根被纳入参数更新。虽然这些方法常被近似为二阶方法，但平方根是一个根本差异。在本研究中，我们探究了去掉平方根（即增强其二阶动机）后自适应方法的行为变化。令人惊讶的是，我们发现这种无平方根的自适应方法在卷积架构上缩小了与SGD的泛化差距，同时在transformer上保持了其带平方根对应方法的性能。二阶视角对于开发非对角方法也有实际益处，这类方法可通过预处理器不变性的概念纳入任意曲率近似。与Shampoo等带平方根的方法相比，无平方根的对应方法在半精度下运行良好且速度快，因为它们不需要数值不稳定的矩阵平方根分解和求逆。总体而言，我们的发现为自适应方法的发展提供了新见解，并提出了关于适应性在其成功中被忽视的作用的重要问题。（实验代码：https://github.com/yorkerlin/remove-the-square-root 优化器代码：https://github.com/f-dangel/sirfshampoo）
英文摘要：
Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers. Their diagonal preconditioner is based on the gradient outer product which is incorporated into the parameter update via a square root. While these methods are often motivated as approximate second-order methods, the square root represents a fundamental difference. In this work, we investigate how the behavior of adaptive methods changes when we remove the root, i.e., strengthen their second-order motivation. Surprisingly, we find that such square-root-free adaptive methods close the generalization gap to SGD on convolutional architectures, while maintaining their root-based counterpart's performance on transformers. The second-order perspective also has practical benefits for developing non-diagonal methods that can incorporate arbitrary curvature approximations through the concept of preconditioner invariance. In contrast to root-based methods like Shampoo, root-free counterparts work well and fast with half-precision since they do not require numerically unstable matrix root decompositions and inversions. Overall, our findings provide new insights into the development of adaptive methods and raise important questions regarding the overlooked role of adaptivity in their success. (experiment code: https://github.com/yorkerlin/remove-the-square-root optimizer code: https://github.com/f-dangel/sirfshampoo)
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
