arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.07484cs.LG
收藏

在可分离随机近似框架下的在线学习

Online Learning Under A Separable Stochastic Approximation Framework

Fuzhou University(福州大学);College of Computer and Data Science(计算机与数据科学学院)

Min Gan, Xiang-xiang Su, Guang-yong Chen, Jing Chen

AI总结:

提出一种可分离随机近似框架下的在线学习算法,利用RLS更新线性参数、SGD更新非线性参数,在非凸情况下证明了全局收敛性,实验显示其显著加速收敛且对学习率不敏感。

摘要:

我们提出了一种在可分离随机近似框架下针对一类机器学习模型的在线学习算法。我们的核心思想在于观察到模型中的某些参数比其他参数更容易优化。在本文中,我们重点关注具有部分线性参数的模型,这在机器学习中很常见。在所提出算法的一个常规步骤中,线性参数通过递归最小二乘(RLS)算法进行更新,这等价于一种随机牛顿方法;然后,基于更新后的线性参数,非线性参数通过随机梯度下降法(SGD)进行更新。所提出的算法可以被理解为块坐标梯度下降方法的随机近似版本,其中一部分参数通过二阶 SGD 方法更新,而另一部分参数通过一阶 SGD 更新。我们针对非凸情况,以一阶最优性条件的期望违备度形式,建立了所提出在线算法的全局收敛性。数值实验表明,与其他流行的学习算法相比,所提出的方法显著加速了收敛,并产生了更鲁棒的训练和测试性能。此外,我们的算法对学习率不太敏感,并且优于最近提出的 slimTrain 算法(Newman 等,2022)。代码已上传至 GitHub 供验证。

英文摘要:

We propose an online learning algorithm for a class of machine learning models under a separable stochastic approximation framework. The essence of our idea lies in the observation that certain parameters in the models are easier to optimize than others. In this paper, we focus on models where some parameters have a linear nature, which is common in machine learning. In one routine of the proposed algorithm, the linear parameters are updated by the recursive least squares (RLS) algorithm, which is equivalent to a stochastic Newton method; then, based on the updated linear parameters, the nonlinear parameters are updated by the stochastic gradient method (SGD). The proposed algorithm can be understood as a stochastic approximation version of block coordinate gradient descent approach in which one part of the parameters is updated by a second-order SGD method while the other part is updated by a first-order SGD. Global convergence of the proposed online algorithm for non-convex cases is established in terms of the expected violation of a first-order optimality condition. Numerical experiments show that the proposed method accelerates convergence significantly and produces more robust training and test performance when compared to other popular learning algorithms. Moreover, our algorithm is less sensitive to the learning rate and outperforms the recently proposed slimTrain algorithm (Newman et al., 2022). The code has been uploaded to GitHub for validation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑