arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.00352cs.LGastro-ph.IMhep-thmath.OCstat.ML
收藏

改进面向机器学习的能量守恒下降法:理论与实践

Improving Energy Conserving Descent for Machine Learning: Theory and Practice

Stanford Institute for Theoretical Physics, Stanford University(斯坦福大学理论物理研究所,斯坦福大学)

G. Bruno De Luca, Alice Gatti, Eva Silverstein

AI总结:

本文发展 ECD 优化理论并提出 ECDSep 算法,通过能量守恒混沌动力系统改进动力学与混沌诱导机制,在多类机器学习任务上达到优于或持平 SGD、Adam、AdamW 的效果。

摘要:

我们发展了能量守恒下降(Energy Conserving Descent,ECD)理论,并提出了 ECDSep,这是一种基于梯度的优化算法,能够处理凸和非凸优化问题。该方法建立在新颖的 ECD 框架之上,将优化视为一个适当的混沌能量守恒动力系统的物理演化,从而能够对结果分布进行解析控制;即使对于没有对称性的一般高维问题,该分布也以低损失为主。与这一思想此前的实现相比,我们利用理论控制改进了动力学和诱导混沌的要素,在提升性能的同时,简化了面向不同问题类别的优化算法超参数调节。我们在广泛的机器学习问题上与 SGD、Adam 和 AdamW 等流行优化方法进行了实证比较,发现其性能与每项任务中这些方法的最佳者相比具有竞争力或有所提升。我们还指出了分析中的局限性,这些局限性指明了进一步改进的可能性。

英文摘要:

We develop the theory of Energy Conserving Descent (ECD) and introduce ECDSep, a gradient-based optimization algorithm able to tackle convex and non-convex optimization problems. The method is based on the novel ECD framework of optimization as physical evolution of a suitable chaotic energy-conserving dynamical system, enabling analytic control of the distribution of results - dominated at low loss - even for generic high-dimensional problems with no symmetries. Compared to previous realizations of this idea, we exploit the theoretical control to improve both the dynamics and chaos-inducing elements, enhancing performance while simplifying the hyper-parameter tuning of the optimization algorithm targeted to different classes of problems. We empirically compare with popular optimization methods such as SGD, Adam and AdamW on a wide range of machine learning problems, finding competitive or improved performance compared to the best among them on each task. We identify limitations in our analysis pointing to possibilities for additional improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑