arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.04707cs.LGcs.AI
收藏

通过优化梯度下降增强深度学习:连接数值方法与神经网络训练

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Johns Hopkins University(约翰斯·霍普金斯大学);Columbia University(哥伦比亚大学);Washington University in St. Louis(圣路易斯华盛顿大学);New York University(纽约大学);University of Chicago(芝加哥大学)

Yuhan Ma, Dan Sun, Erdi Gao, Ningjing Sang, Iris Li, Guanming Huang

AI总结:

本文面向深度学习训练,借鉴数值优化方法改进 SGD 优化器,并通过多类任务实验验证其在可解释性、准确性和有效性上的提升。

摘要:

优化理论是实现系统最优性能的关键科学工具,其起源可追溯至经济应用,用于确定最大化收益的最佳投资策略。几个世纪以来,从古希腊的几何学探索,到 Newton 和 Leibniz 对微积分的贡献,优化理论取得了显著进展。Lagrange、Cauchy 和 von Neumann 等科学家持续不懈的工作进一步巩固了其发展。现代时期,优化理论的应用出现了前所未有的扩展,尤其是随着计算机科学的发展,更复杂的计算实践得以实现,并在工程、决策分析和运筹学中得到广泛应用。本文深入探讨优化理论与深度学习之间的深刻关系,强调优化问题在深度学习中的普遍存在。我们研究了作为神经网络优化基石的梯度下降算法及其变体。该章节介绍了对 SGD 优化器的一项增强,其灵感来自数值优化方法,旨在提升可解释性和准确性。我们在多种深度学习任务上的实验证实了改进后算法的有效性。论文最后强调,优化理论将持续发展,并在解决复杂问题、增强计算能力以及为更好的政策决策提供依据方面发挥日益重要的作用。

英文摘要:

Optimization theory serves as a pivotal scientific instrument for achieving optimal system performance, with its origins in economic applications to identify the best investment strategies for maximizing benefits. Over the centuries, from the geometric inquiries of ancient Greece to the calculus contributions by Newton and Leibniz, optimization theory has significantly advanced. The persistent work of scientists like Lagrange, Cauchy, and von Neumann has fortified its progress. The modern era has seen an unprecedented expansion of optimization theory applications, particularly with the growth of computer science, enabling more sophisticated computational practices and widespread utilization across engineering, decision analysis, and operations research. This paper delves into the profound relationship between optimization theory and deep learning, highlighting the omnipresence of optimization problems in the latter. We explore the gradient descent algorithm and its variants, which are the cornerstone of optimizing neural networks. The chapter introduces an enhancement to the SGD optimizer, drawing inspiration from numerical optimization methods, aiming to enhance interpretability and accuracy. Our experiments on diverse deep learning tasks substantiate the improved algorithm's efficacy. The paper concludes by emphasizing the continuous development of optimization theory and its expanding role in solving intricate problems, enhancing computational capabilities, and informing better policy decisions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑