一种用于深度神经网络训练的两阶段子空间信赖域方法
A Two-Stage Subspace Trust Region Approach for Deep Neural Network Training
摘要:
在本文中,我们开发了一种新的二阶方法来训练前馈神经网络。在每次迭代中,我们在低维子空间中构造成本函数的二次近似。我们通过两个阶段的过程来最小化信赖域内的近似:首先在嵌入的正曲率子空间内,然后是梯度下降步骤。这种方法导致快速的目标函数衰减,防止收敛到鞍点,并消除了手动调整参数的需要。我们在基准数据集上显示了该算法的良好性能。
英文摘要:
In this paper, we develop a novel second-order method for training feed-forward neural nets. At each iteration, we construct a quadratic approximation to the cost function in a low-dimensional subspace. We minimize this approximation inside a trust region through a two-stage procedure: first inside the embedded positive curvature subspace, followed by a gradient descent step. This approach leads to a fast objective function decay, prevents convergence to saddle points, and alleviates the need for manually tuning parameters. We show the good performance of the proposed algorithm on benchmark datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。