arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.00631cs.LGmath.OC
收藏

双向审视:面向自适应与非自适应动量优化器的新型双重指数移动平均

Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers

Wuhan University(武汉大学);Hubei Luojia Laboratory(湖北珞珈实验室);Shanghai Jiao Tong University(上海交通大学)

Yineng Chen, Zuchao Li, Lefei Zhang, Bo Du, Hai Zhao

AI总结:

结合优化器的后向与前瞻特性,提出基于双重指数移动平均和动态lookahead策略的Admeta优化器框架,及其变体AdmetaR和AdmetaS,实验与理论证明其优于基础优化器且具收敛性。

摘要:

优化器是深度学习成功的关键组件,它引导神经网络根据训练集上的损失来更新参数。SGD和Adam是两种经典且有效的优化器,研究人员在此基础上提出了许多变体,如SGDM和RAdam。本文创新性地结合了优化器算法的后向审视与前瞻特性,提出了一种新型的Admeta(一种双重指数移动平均用于自适应与非自适应动量)优化器框架。在后向审视部分,我们受股票市场指标的启发,提出了一种DEMA变体方案,以替代常见的指数移动平均方案。而在前瞻部分,我们提出了一种动态lookahead策略,该策略渐近趋近于设定值,在早期保持速度并在最终阶段保持高收敛性能。基于这一思想,我们提供了两种优化器实现,AdmetaR和AdmetaS,前者基于RAdam,后者基于SGDM。通过在多种任务上的广泛实验,我们发现所提出的Admeta优化器优于我们的基础优化器,并展现出相对于最近提出的竞争优化器的优势。我们还提供了这两种算法的理论证明,验证了所提出的Admeta的收敛性。

英文摘要:

Optimizer is an essential component for the success of deep learning, which guides the neural network to update the parameters according to the loss on the training set. SGD and Adam are two classical and effective optimizers on which researchers have proposed many variants, such as SGDM and RAdam. In this paper, we innovatively combine the backward-looking and forward-looking aspects of the optimizer algorithm and propose a novel \textsc{Admeta} (\textbf{A} \textbf{D}ouble exponential \textbf{M}oving averag\textbf{E} \textbf{T}o \textbf{A}daptive and non-adaptive momentum) optimizer framework. For backward-looking part, we propose a DEMA variant scheme, which is motivated by a metric in the stock market, to replace the common exponential moving average scheme. While in the forward-looking part, we present a dynamic lookahead strategy which asymptotically approaches a set value, maintaining its speed at early stage and high convergence performance at final stage. Based on this idea, we provide two optimizer implementations, \textsc{AdmetaR} and \textsc{AdmetaS}, the former based on RAdam and the latter based on SGDM. Through extensive experiments on diverse tasks, we find that the proposed \textsc{Admeta} optimizer outperforms our base optimizers and shows advantages over recently proposed competitive optimizers. We also provide theoretical proof of these two algorithms, which verifies the convergence of our proposed \textsc{Admeta}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑