基于ODE分裂的随机梯度算法
Stochastic gradient algorithms from ODE splitting perspective
摘要：
我们对随机最优化提出了一种不同的观点，这种观点可以追溯到常微分方程近似解的分裂方案。在这项工作中，我们提供了一个随机梯度下降方法和一阶分裂格式之间的联系。我们考虑了受机器学习应用启发的分裂的特殊情况，得到了一个新的全局分裂误差的上界，证明了Kaczmarz方法是线性最小二乘问题单位批SGD分裂方案的极限情况.我们通过系统的实证研究来支持我们的研究结果，这些研究表明，局部问题的更准确的解决方案会导致步长鲁棒性，并在softmax回归问题上提供更好的时间收敛和迭代。
英文摘要：
We present a different view on stochastic optimization, which goes back to the splitting schemes for approximate solutions of ODE. In this work, we provide a connection between stochastic gradient descent approach and first-order splitting scheme for ODE. We consider the special case of splitting, which is inspired by machine learning applications and derive a new upper bound on the global splitting error for it. We present, that the Kaczmarz method is the limit case of the splitting scheme for the unit batch SGD for linear least squares problem. We support our findings with systematic empirical studies, which demonstrates, that a more accurate solution of local problems leads to the stepsize robustness and provides better convergence in time and iterations on the softmax regression problem.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
