随机最优化的自适应抽样策略
Adaptive Sampling Strategies for Stochastic Optimization
摘要:
在本文中,我们提出了一个随机最优化方法,自适应地控制用于梯度近似计算的样本量。与其他方差降低技术,要么需要额外的存储或定期计算的全梯度,所提出的方法减少方差,根据需要增加样本量。增加样本量的决定由内积检验决定,该检验确保搜索方向是具有高概率的下降方向。我们证明了内积检验改进了著名的范数检验,并且可以作为一个算法的基础,该算法在非凸函数上是全局收敛的,并且在强凸函数上具有全局线性收敛速率。对逻辑斯谛回归问题的数值实验表明了该算法的性能。
英文摘要:
In this paper, we propose a stochastic optimization method that adaptively controls the sample size used in the computation of gradient approximations. Unlike other variance reduction techniques that either require additional storage or the regular computation of full gradients, the proposed method reduces variance by increasing the sample size as needed. The decision to increase the sample size is governed by an inner product test that ensures that search directions are descent directions with high probability. We show that the inner product test improves upon the well known norm test, and can be used as a basis for an algorithm that is globally convergent on nonconvex functions and enjoys a global linear rate of convergence on strongly convex functions. Numerical experiments on logistic regression problems illustrate the performance of the algorithm.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。