Switch EMA:实现更优平坦度与锐度的免费午餐
Switch EMA: A Free Lunch for Better Flatness and Sharpness
AI总结:
针对现有权重平均正则化方法存在性能下降或需额外测试计算的问题,提出仅需一行修改的Switch EMA方法,可在平坦度与锐度间实现更优权衡,经多任务多数据集验证,能无额外成本地提升DNN性能并加快收敛。
摘要:
指数移动平均(Exponential Moving Average,EMA)是一种被广泛使用的权重平均(weight averaging,WA)正则化方法,用于在深度神经网络(DNN)优化中以无额外成本的方式学习平坦最优解,从而提升泛化能力。尽管现有WA方法能实现更优的平坦度,但它们可能会导致最终性能变差,或是需要额外的测试时计算。本研究仅通过一行修改就释放了EMA的全部潜力,即在每个epoch后将EMA参数切换回原始模型,该方法被命名为Switch EMA(SEMA)。我们从理论和实证两方面证明,SEMA能够帮助DNN达到在平坦度与锐度之间实现更优权衡的泛化最优解。为验证SEMA的有效性,我们在视觉和语言数据集上开展了判别、生成和回归任务的对比实验,涵盖图像分类、自监督学习、目标检测与分割、图像生成、视频预测、属性回归以及语言建模。结合主流优化器和网络的综合结果表明,SEMA是DNN训练的“免费午餐”,既能提升性能,又能加快收敛速度。
英文摘要:
Exponential Moving Average (EMA) is a widely used weight averaging (WA) regularization to learn flat optima for better generalizations without extra cost in deep neural network (DNN) optimization. Despite achieving better flatness, existing WA methods might fall into worse final performances or require extra test-time computations. This work unveils the full potential of EMA with a single line of modification, i.e., switching the EMA parameters to the original model after each epoch, dubbed as Switch EMA (SEMA). From both theoretical and empirical aspects, we demonstrate that SEMA can help DNNs to reach generalization optima that better trade-off between flatness and sharpness. To verify the effectiveness of SEMA, we conduct comparison experiments with discriminative, generative, and regression tasks on vision and language datasets, including image classification, self-supervised learning, object detection and segmentation, image generation, video prediction, attribute regression, and language modeling. Comprehensive results with popular optimizers and networks show that SEMA is a free lunch for DNN training by improving performances and boosting convergence speeds.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。