arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.13677math.NAcs.LGcs.NA
收藏

神经网络训练中的并行信赖域方法:超越传统方法

Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods

Università della Svizzera italiana(瑞士意大利语区大学);UniDistance Suisse(瑞士远程大学);Brown University(布朗大学)

Ken Trotti, Samuel A. Cruz Alegría, Alena Kopaničáková, Rolf Krause

AI总结:

提出一种基于加性域分解的并行信赖域(APTS)神经网络训练方法,确保全局收敛且免超参数调整,并通过与SGD、Adam等对比实验展示其能力与局限。

摘要:

我们提出使用一种新颖的“加性预处理信赖域策略”(APTS)变体来训练神经网络(NN)。所提出的方法基于一种可并行的加性域分解方法,应用于神经网络的参数。建立在TR框架之上,APTS方法确保全局收敛到极小值。此外,它消除了对计算昂贵的超参数调整的需求,因为TR算法在每次迭代中自动确定步长。我们通过一系列数值实验展示了所提出的APTS训练方法的能力、优势和局限性。所呈现的数值研究包括与广泛使用的训练方法(如SGD、Adam、LBFGS和标准TR方法)的比较。

英文摘要:

We propose to train neural networks (NNs) using a novel variant of the ``Additively Preconditioned Trust-region Strategy'' (APTS). The proposed method is based on a parallelizable additive domain decomposition approach applied to the neural network's parameters. Built upon the TR framework, the APTS method ensures global convergence towards a minimizer. Moreover, it eliminates the need for computationally expensive hyper-parameter tuning, as the TR algorithm automatically determines the step size in each iteration. We demonstrate the capabilities, strengths, and limitations of the proposed APTS training method by performing a series of numerical experiments. The presented numerical study includes a comparison with widely used training methods such as SGD, Adam, LBFGS, and the standard TR method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑