arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.15314cs.LG
收藏

减少深度学习优化中的偏差:RSGDM方法

Reducing Bias in Deep Learning Optimization: The RSGDM Approach

Stevens Institute of Technology(史蒂文斯理工学院);University of California, Berkeley(加州大学伯克利分校);Chinese University of Hong Kong(香港中文大学);University of California, Irvine(加州大学欧文分校);Illinois Institute of Technology(伊利诺伊理工学院);Southern Methodist University(南卫理公会大学)

Honglin Qin, Hongye Zheng, Bingxing Wang, Zhizhong Wu, Bingyao Liu, Yuanfang Yang

AI总结:

本文针对SGDM等优化器使用指数移动平均估计梯度存在偏差和滞后的问题,提出基于微分校正的RSGDM算法,实验证明其在CIFAR数据集上的收敛精度优于SGDM。

摘要:

目前,广泛使用的一阶深度学习优化器包括非自适应学习率优化器和自适应学习率优化器。前者以 SGDM(带动量的随机梯度下降)为代表,后者以 Adam 为代表。这两种方法都使用指数移动平均来估计总体梯度。然而,使用指数移动平均估计总体梯度存在偏差且具有滞后性。本文提出了一种基于微分校正的 RSGDM 算法。我们的贡献主要有三个方面:1)分析了 SGDM 算法中指数移动平均带来的偏差和滞后。2)使用微分估计项来校正 SGDM 算法中的偏差和滞后,提出了 RSGDM 算法。3)在 CIFAR 数据集上的实验证明,我们的 RSGDM 算法在收敛精度方面优于 SGDM 算法。

英文摘要:

Currently, widely used first-order deep learning optimizers include non-adaptive learning rate optimizers and adaptive learning rate optimizers. The former is represented by SGDM (Stochastic Gradient Descent with Momentum), while the latter is represented by Adam. Both of these methods use exponential moving averages to estimate the overall gradient. However, estimating the overall gradient using exponential moving averages is biased and has a lag. This paper proposes an RSGDM algorithm based on differential correction. Our contributions are mainly threefold: 1) Analyze the bias and lag brought by the exponential moving average in the SGDM algorithm. 2) Use the differential estimation term to correct the bias and lag in the SGDM algorithm, proposing the RSGDM algorithm. 3) Experiments on the CIFAR datasets have proven that our RSGDM algorithm is superior to the SGDM algorithm in terms of convergence accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑