arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.12979cs.LG
收藏

一种等距随机优化器

An Isometric Stochastic Optimizer

Jacob Jackson

AI总结:

本文解释了Adam优化器成功的原因,据此提出参数更新范数不受输入输出线性变换影响的Iso优化器,以及可迁移Adam超参数的IsoAdam变体,实验显示IsoAdam训练小型Transformer的速度快于Adam。

摘要:

Adam优化器是深度学习应用中的标准选择。本文提出了Adam成功的一个简单解释:它使每个参数的步长与其他参数的范数无关。基于这一原理,本文推导了新型优化器Iso,该优化器可使参数更新的范数在对其输入和输出施加任意线性变换时保持不变。本文还开发了Iso的变体IsoAdam,可从Adam迁移最优超参数,并证明在训练小型Transformer时,IsoAdam的速度优于Adam。

英文摘要:

The Adam optimizer is the standard choice in deep learning applications. I propose a simple explanation of Adam's success: it makes each parameter's step size independent of the norms of the other parameters. Based on this principle I derive Iso, a new optimizer which makes the norm of a parameter's update invariant to the application of any linear transformation to its inputs and outputs. I develop a variant of Iso called IsoAdam that allows optimal hyperparameters to be transferred from Adam, and demonstrate that IsoAdam obtains a speedup over Adam when training a small Transformer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑