arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.09566cs.LGmath.OC
收藏

深度学习中的优化方法:综合综述

Optimization Methods in Deep Learning: A Comprehensive Overview

David Shulman

AI总结:

本文综述深度学习训练中的一阶、动量及自适应梯度优化方法,分析优化挑战及初始化与归一化技术,并给出面向任务和数据集的选型建议。

摘要:

近年来,深度学习在图像识别、自然语言处理和语音识别等多个领域取得了显著成功。深度学习的有效性在很大程度上取决于用于训练深度神经网络的优化方法。本文概述了一阶优化方法,如 Stochastic Gradient Descent、Adagrad、Adadelta 和 RMSprop,以及近期基于动量和自适应梯度的方法,如 Nesterov accelerated gradient、Adam、Nadam、AdaMax 和 AMSGrad。我们还讨论了深度学习中与优化相关的挑战,并探讨了应对这些挑战的技术,包括权重初始化、batch normalization 和 layer normalization。最后,我们针对不同深度学习任务和数据集给出了选择优化方法的建议。本文可作为深度学习优化方法的综合指南,并可供该领域的研究人员和从业者参考。

英文摘要:

In recent years, deep learning has achieved remarkable success in various fields such as image recognition, natural language processing, and speech recognition. The effectiveness of deep learning largely depends on the optimization methods used to train deep neural networks. In this paper, we provide an overview of first-order optimization methods such as Stochastic Gradient Descent, Adagrad, Adadelta, and RMSprop, as well as recent momentum-based and adaptive gradient methods such as Nesterov accelerated gradient, Adam, Nadam, AdaMax, and AMSGrad. We also discuss the challenges associated with optimization in deep learning and explore techniques for addressing these challenges, including weight initialization, batch normalization, and layer normalization. Finally, we provide recommendations for selecting optimization methods for different deep learning tasks and datasets. This paper serves as a comprehensive guide to optimization methods in deep learning and can be used as a reference for researchers and practitioners in the field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑