Optimization Methods in Deep Learning: A Comprehensive Overview
David Shulman
备注:
We have decided to withdraw our manuscript 'Optimization Methods in Deep Learning: A Comprehensive Overview' from arXiv as we are now pursuing different research directions. We apologize for any inconvenience and appreciate your understanding
In recent years, deep learning has achieved remarkable success in various fields such as image recognition, natural language processing, and speech recognition. The effectiveness of deep learning largely depends on the optimization methods used to train deep neural networks. In this paper, we provide an overview of first-order optimization methods such as Stochastic Gradient Descent, Adagrad, Adadelta, and RMSprop, as well as recent momentum-based and adaptive gradient methods such as Nesterov accelerated gradient, Adam, Nadam, AdaMax, and AMSGrad. We also discuss the challenges associated with optimization in deep learning and explore techniques for addressing these challenges, including weight initialization, batch normalization, and layer normalization. Finally, we provide recommendations for selecting optimization methods for different deep learning tasks and datasets. This paper serves as a comprehensive guide to optimization methods in deep learning and can be used as a reference for researchers and practitioners in the field.
Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview
Fraunhofer Institut für integrierte Schaltungen(弗劳恩霍夫集成电路研究所);Ludwig-Maximilians-Universität München(慕尼黑大学);Technische Universität Dortmund(多特蒙德工业大学);Universidad Pontificia Comillas(康普卢滕塞大学);Warwick Business School(华威商学院)
Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview
Peking University(北京大学);Nankai University(南开大学);Beijing Institute of Technology(北京理工大学);University of Leeds(利兹大学);Harbin University of Commerce(哈尔滨商业大学)
An Overview and Experimental Study of Learning-based Optimization Algorithms for Vehicle Routing Problem
Central South University(中南大学);National University of Defense Technology(国防科技大学);University of Alberta(阿尔伯塔大学);King Abdulaziz University(阿卜杜勒阿齐兹国王大学);Polish Academy of Sciences(波兰科学院)
An Overview of Machine Learning-Enabled Optimization for Reconfigurable Intelligent Surfaces-Aided 6G Networks: From Reinforcement Learning to Large Language Models
School of Computer Science, McGill University(麦吉尔大学计算机科学学院)
A Comprehensive Overview on 5G-and-Beyond Networks with UAVs: From Communications to Sensing and Intelligence
University of Macau(澳门大学);Future Network of Intelligence Institute(未来网络智能研究院);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));Southeast University(东南大学);Purple Mountain Laboratories(紫金山实验室);University of New South Wales(新南威尔士大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);Friedrich-Alexander-University Erlangen-Nurnberg(弗里德里希-亚历山大-埃尔朗根-纽伦堡大学);University of California, Irvine(加州大学欧文分校)
A Comprehensive Data-centric Overview of Federated Graph Learning
Beijing Institute of Technology(北京理工大学);Sun Yat-sen University(中山大学);Peking University(北京大学);Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学学院);Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心)
Advanced Unsupervised Learning: A Comprehensive Overview of Multi-View Clustering Techniques
Universidad Internacional de La Rioja (UNIR)(拉里奥哈国际大学);University of the Basque Country(巴斯克大学);IKERBASQUE, Basque Foundation for Science(伊克尔巴斯克,巴斯克科学基金会)