arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.01036cs.LGcs.AI
收藏

eagle:基于早期近似梯度的学习率估计器

eagle: early approximated gradient based learning rate estimator

Center for Information and Computer Science, School of Science for Open and Environmental Systems, Keio University(庆应义塾大学开放与环境系统科学学院信息与计算机科学中心)

Takumi Fujimoto, Hiroaki Nishi

AI总结:

提出EAGLE优化方法,利用连续步骤的参数与梯度变化及局部曲率估计最优参数,并引入Adam与EAGLE间的自适应切换机制以增强稳定性,实验证明其能以更少的epoch加速训练损失收敛。

摘要:

我们提出了EAGLE更新规则,这是一种通过同时利用当前和前一步的参数与梯度值来加速训练早期损失收敛的新颖优化方法。该更新算法通过计算连续训练步骤间参数和梯度的变化,并利用由这些变化推导出的损失景观局部曲率来估计最优参数。然而,该更新规则具有潜在的不稳定性,为解决此问题,我们引入了一种自适应切换机制,在Adam和EAGLE更新规则之间动态选择以增强训练稳定性。在标准基准数据集上的实验表明,结合了该新颖更新规则与切换机制的EAGLE优化器,与传统优化方法相比,能够以更少的epoch实现快速的训练损失收敛。

英文摘要:

We propose EAGLE update rule, a novel optimization method that accelerates loss convergence during the early stages of training by leveraging both current and previous step parameter and gradient values. The update algorithm estimates optimal parameters by computing the changes in parameters and gradients between consecutive training steps and leveraging the local curvature of the loss landscape derived from these changes. However, this update rule has potential instability, and to address that, we introduce an adaptive switching mechanism that dynamically selects between Adam and EAGLE update rules to enhance training stability. Experiments on standard benchmark datasets demonstrate that EAGLE optimizer, which combines this novel update rule with the switching mechanism achieves rapid training loss convergence with fewer epochs, compared to conventional optimization methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑