AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimate
AdamNX:一种基于二阶矩估计的新型指数衰减机制的Adam改进算法
机构 * School of Information Management and Mathematics, Jiangxi University of Finance and Economics(江西财经大学信息管理与数学学院) ; School of Mathematics and Computer Science, Nanchang University(南昌大学数学与计算机科学学院) ; Institute of Metaverse, Nanchang University(南昌大学元宇宙研究院) ; Jiangxi Provincial Key Laboratory of Virtual Reality(江西省虚拟现实重点实验室)
专题命中 领域大模型 :large language model(abstract);language model(abstract);分类 cs.LG
AI总结 研究Adam中二阶矩估计的指数衰减机制,提出AdamNX及新衰减率,使更新在训练平稳阶段接近动量-SGD行为,通过图像分类等任务实验报告结果,代码已开源。
Comments 29 pages, 6 figures, 12 tables