arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.07488cs.LG
收藏

通过预条件子对角化改进自适应矩优化

Improving Adaptive Moment Optimization via Preconditioner Diagonalization

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Son Nguyen, Bo Liu, Lizhang Chen, Qiang Liu

AI总结:

针对自适应优化方法因未充分利用梯度协方差信息导致收敛慢的问题,提出通过可逆变换将结构化预条件子矩阵映射至新空间实现近似对角化的方法,显著提升收敛速度,对LLaMA实现2倍加速且可与Adafactor集成。

摘要:

近年来,Adam及其变体等现代自适应优化方法已成为深度学习中应用最广泛的工具。这些算法提供了基于梯度统计估计动态调整更新步长的自动机制。与随机梯度下降(Stochastic Gradient Descent)等传统算法相比,这些自适应方法通常对模型规模和超参数调优更具鲁棒性。然而,这些方法使用的梯度统计往往未充分利用梯度协方差信息,导致参数空间某些方向的更新次优,可能减慢收敛速度。在本研究中,我们以结构化预条件子矩阵的形式跟踪此类协方差统计。与其他研究不同,我们的方法不直接近似估计该矩阵,而是实现一种可逆变换,将预条件子矩阵映射到一个新空间,使其近似对角化。这使得在变换空间中对预条件子矩阵进行对角近似,具有若干计算优势。实证结果表明,我们的方法可显著提升现代自适应优化器的收敛速度。值得注意的是,对于LLaMA等大型语言模型,与基线Adam相比,我们可实现2倍的加速。此外,我们的方法可与Adafactor等内存高效优化器集成,以管理计算开销。

英文摘要:

Modern adaptive optimization methods, such as Adam and its variants, have emerged as the most widely used tools in deep learning over recent years. These algorithms offer automatic mechanisms for dynamically adjusting the update step based on estimates of gradient statistics. Compared to traditional algorithms like Stochastic Gradient Descent, these adaptive methods are typically more robust to model scale and hyperparameter tuning. However, the gradient statistics employed by these methods often do not leverage sufficient gradient covariance information, leading to suboptimal updates in certain directions of the parameter space and potentially slower convergence. In this work, we keep track of such covariance statistics in the form of a structured preconditioner matrix. Unlike other works, our approach does not apply direct approximations to estimate this matrix. We instead implement an invertible transformation that maps the preconditioner matrix into a new space where it becomes approximately diagonal. This enables a diagonal approximation of the preconditioner matrix in the transformed space, offering several computational advantages. Empirical results show that our approach can substantially enhance the convergence speed of modern adaptive optimizers. Notably, for large language models like LLaMA, we can achieve a speedup of 2x compared to the baseline Adam. Additionally, our method can be integrated with memory-efficient optimizers like Adafactor to manage computational overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑