神经网络训练中的并行信赖域方法:超越传统方法
Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods
AI总结:
提出一种基于加性域分解的并行信赖域(APTS)神经网络训练方法,确保全局收敛且免超参数调整,并通过与SGD、Adam等对比实验展示其能力与局限。
摘要:
我们提出使用一种新颖的“加性预处理信赖域策略”(APTS)变体来训练神经网络(NN)。所提出的方法基于一种可并行的加性域分解方法,应用于神经网络的参数。建立在TR框架之上,APTS方法确保全局收敛到极小值。此外,它消除了对计算昂贵的超参数调整的需求,因为TR算法在每次迭代中自动确定步长。我们通过一系列数值实验展示了所提出的APTS训练方法的能力、优势和局限性。所呈现的数值研究包括与广泛使用的训练方法(如SGD、Adam、LBFGS和标准TR方法)的比较。
英文摘要:
We propose to train neural networks (NNs) using a novel variant of the ``Additively Preconditioned Trust-region Strategy'' (APTS). The proposed method is based on a parallelizable additive domain decomposition approach applied to the neural network's parameters. Built upon the TR framework, the APTS method ensures global convergence towards a minimizer. Moreover, it eliminates the need for computationally expensive hyper-parameter tuning, as the TR algorithm automatically determines the step size in each iteration. We demonstrate the capabilities, strengths, and limitations of the proposed APTS training method by performing a series of numerical experiments. The presented numerical study includes a comparison with widely used training methods such as SGD, Adam, LBFGS, and the standard TR method.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。