插值网络的随机束方法
A Stochastic Bundle Method for Interpolating Networks
摘要：
我们提出了一种新的方法来训练能够插值的深度神经网络，即将经验损失驱动到零。在每一次迭代中，我们的方法构造了学习目标的随机近似。近似，称为丛，是线性函数的逐点最大值。我们的捆绑包包含一个常数函数，它降低了经验损失。这使我们能够计算自动自适应学习率，从而提供准确的解决方案。此外，我们的捆绑包还包括在当前时间点计算的线性近似和DNN参数的其他线性估计。这些额外近似的使用使我们的方法对其超参数更加鲁棒。基于其理想的经验性质，我们将我们的方法称为Bundle Optimisation for Robust and Accurate Training（BORAT）。为了操作BORAT，我们设计了一种新的算法，有效地优化束近似在每次迭代。我们建立了理论上的收敛BORAT在凸和非凸设置。使用标准的公开数据集，我们提供了一个彻底的比较BORAT其他单超参数优化算法。我们的实验表明，BORAT匹配这些方法的最先进的泛化性能，是最强大的。
英文摘要：
We propose a novel method for training deep neural networks that are capable of interpolation, that is, driving the empirical loss to zero. At each iteration, our method constructs a stochastic approximation of the learning objective. The approximation, known as a bundle, is a pointwise maximum of linear functions. Our bundle contains a constant function that lower bounds the empirical loss. This enables us to compute an automatic adaptive learning rate, thereby providing an accurate solution. In addition, our bundle includes linear approximations computed at the current iterate and other linear estimates of the DNN parameters. The use of these additional approximations makes our method significantly more robust to its hyperparameters. Based on its desirable empirical properties, we term our method Bundle Optimisation for Robust and Accurate Training (BORAT). In order to operationalise BORAT, we design a novel algorithm for optimising the bundle approximation efficiently at each iteration. We establish the theoretical convergence of BORAT in both convex and non-convex settings. Using standard publicly available data sets, we provide a thorough comparison of BORAT to other single hyperparameter optimisation algorithms. Our experiments demonstrate BORAT matches the state-of-the-art generalisation performance for these methods and is the most robust.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
