arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.01029cs.LG
收藏

关于抑制Adam自适应步长范围以提升泛化性能的研究

On Suppressing Range of Adaptive Stepsizes of Adam to Improve Generalisation Performance

School of Electrical and Data Engineering, University of Technology, Sydney（悉尼科技大学电气与数据工程学院）

Guoqiang Zhang

备注：
Accepted by ECML 2024. arXiv admin note: substantial text overlap with arXiv:2203.13273
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究针对Adam泛化性能问题，提出SET-Adam算法，通过对二阶动量v_t执行逐层向下缩放、epsilon嵌入、向下平移三项操作抑制自适应步长范围，在多个NLP、图像分类及生成任务上优于或匹配现有自适应优化器性能。

摘要：

近期诸多自适应优化器通过本质上降低自适应步长的方差，使其更接近带动量的SGD，来改善Adam的泛化性能。遵循这一思路，我们利用逐层梯度统计量来抑制Adam自适应步长的范围。具体而言，在每次迭代中，我们提出在使用二阶动量v_t更新深度神经网络（DNN）模型前，对其执行三项连续操作：（1）向下缩放；（2）epsilon嵌入；（3）向下平移。由此得到的算法被称为SET-Adam，其中SET是这三项操作的缩写。对v_t的向下缩放操作是逐层执行的，利用v_t的各层子向量与对应的全1子向量之间的夹角来实现。大量实验结果表明，在针对自然语言处理（NLP）任务训练transformer和LSTM，以及在CIFAR10和CIFAR100数据集上训练VGG和ResNet进行图像分类时，SET-Adam的表现优于八种自适应优化器；而在训练用于图像生成任务的WGAN-GP模型时，其性能与八种自适应方法中的最佳表现相当。此外，在ImageNet数据集上训练ResNet18时，SET-Adam取得了比Adam和AdaBelief更高的验证准确率。

英文摘要：

A number of recent adaptive optimizers improve the generalisation performance of Adam by essentially reducing the variance of adaptive stepsizes to get closer to SGD with momentum. Following the above motivation, we suppress the range of the adaptive stepsizes of Adam by exploiting the layerwise gradient statistics. In particular, at each iteration, we propose to perform three consecutive operations on the second momentum v_t before using it to update a DNN model: (1): down-scaling, (2): epsilon-embedding, and (3): down-translating. The resulting algorithm is referred to as SET-Adam, where SET is a brief notation of the three operations. The down-scaling operation on v_t is performed layerwise by making use of the angles between the layerwise subvectors of v_t and the corresponding all-one subvectors. Extensive experimental results show that SET-Adam outperforms eight adaptive optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR100 while matching the best performance of the eight adaptive methods when training WGAN-GP models for image generation tasks. Furthermore, SET-Adam produces higher validation accuracies than Adam and AdaBelief for training ResNet18 over ImageNet.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑