arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1805.09430cs.CV
收藏

一种用于深度神经网络训练的两阶段子空间信赖域方法

A Two-Stage Subspace Trust Region Approach for Deep Neural Network Training

Taras Shevchenko National University of Kyiv(塔拉斯·舍甫琴科国立基辅大学);Université Paris Est(巴黎东部大学);CentraleSupelec(中央理工-高等电力学院);University Paris-Saclay(巴黎萨克雷大学)

Viacheslav Dudar, Giovanni Chierchia, Emilie Chouzenoux, Jean-Christophe Pesquet, Vladimir Semenov

摘要:

在本文中,我们开发了一种新的二阶方法来训练前馈神经网络。在每次迭代中,我们在低维子空间中构造成本函数的二次近似。我们通过两个阶段的过程来最小化信赖域内的近似:首先在嵌入的正曲率子空间内,然后是梯度下降步骤。这种方法导致快速的目标函数衰减,防止收敛到鞍点,并消除了手动调整参数的需要。我们在基准数据集上显示了该算法的良好性能。

英文摘要:

In this paper, we develop a novel second-order method for training feed-forward neural nets. At each iteration, we construct a quadratic approximation to the cost function in a low-dimensional subspace. We minimize this approximation inside a trust region through a two-stage procedure: first inside the embedded positive curvature subspace, followed by a gradient descent step. This approach leads to a fast objective function decay, prevents convergence to saddle points, and alleviates the need for manually tuning parameters. We show the good performance of the proposed algorithm on benchmark datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑