基于复步长方向导数的二阶神经网络训练
Second-order Neural Network Training Using Complex-step Directional Derivative
摘要：
虽然牛顿法等二阶优化方法的优越性能是众所周知的，但它们在实践中几乎没有用于深度学习，因为组装海森矩阵或计算其逆矩阵对于大规模问题都不可行。现有的二阶方法诉诸于各种对角或低秩近似的海森，这往往无法捕捉必要的曲率信息，以产生实质性的改善。另一方面，当训练变得基于批处理时（即，随机的），噪声二阶信息容易污染训练过程，除非采用昂贵的保护措施。在本文中，我们采用一种数值算法的二阶神经网络的训练。我们解决了实际的障碍，海森计算使用复杂的步骤有限差分（CSFD）-一个数值程序添加一个虚扰动的函数导数计算。CSFD具有高度的鲁棒性、高效性和准确性（与分析结果一样准确）。这种方法允许我们将任何已知的二阶优化方法应用于深度学习训练。在此基础上，我们设计了一个有效的Newton Krylov过程。其关键机制是一旦发现扰动方向就终止随机Krylov迭代，从而避免不必要的计算。在优化过程中，我们监控泰勒展开中的近似误差以调整步长。该策略结合了线搜索和信赖域方法的优点，使我们的方法在保持良好的局部和全局收敛性的同时。我们已经在各种深度学习任务中测试了我们的方法。实验表明，我们的方法优于现有的方法，它往往收敛快一个数量级。我们相信，我们的方法将为深度学习和数值优化带来广泛的新算法。
英文摘要：
While the superior performance of second-order optimization methods such as Newton's method is well known, they are hardly used in practice for deep learning because neither assembling the Hessian matrix nor calculating its inverse is feasible for large-scale problems. Existing second-order methods resort to various diagonal or low-rank approximations of the Hessian, which often fail to capture necessary curvature information to generate a substantial improvement. On the other hand, when training becomes batch-based (i.e., stochastic), noisy second-order information easily contaminates the training procedure unless expensive safeguard is employed. In this paper, we adopt a numerical algorithm for second-order neural network training. We tackle the practical obstacle of Hessian calculation by using the complex-step finite difference (CSFD) -- a numerical procedure adding an imaginary perturbation to the function for derivative computation. CSFD is highly robust, efficient, and accurate (as accurate as the analytic result). This method allows us to literally apply any known second-order optimization methods for deep learning training. Based on it, we design an effective Newton Krylov procedure. The key mechanism is to terminate the stochastic Krylov iteration as soon as a disturbing direction is found so that unnecessary computation can be avoided. During the optimization, we monitor the approximation error in the Taylor expansion to adjust the step size. This strategy combines advantages of line search and trust region methods making our method preserves good local and global convergency at the same time. We have tested our methods in various deep learning tasks. The experiments show that our method outperforms exiting methods, and it often converges one-order faster. We believe our method will inspire a wide-range of new algorithms for deep learning and numerical optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
