次线性收敛算法的计算复杂性
Computational Complexity of Sub-Linear Convergent Algorithms
摘要:
优化用于求解目标函数的机器学习算法已经引起了极大的兴趣。探讨了梯度下降和随机梯度下降等常用算法的优化方法。其中一种方法是通过自适应采样来减小梯度方差,以解决大规模优化的经验风险最小化(ERM)问题。在本文中,我们将探讨如何从一个小样本,然后几何增加它,并使用前一个样本ERM的解决方案来计算新的ERM。这将用次线性收敛的一阶优化算法解决ERM问题,但具有较低的计算复杂性。本文首先从理论上证明了该方法,然后通过两个实验比较了梯度下降与自适应采样的梯度下降和ADAM与自适应采样ADAM在不同的数据集。
英文摘要:
Optimizing machine learning algorithms that are used to solve the objective function has been of great interest. Several approaches to optimize common algorithms, such as gradient descent and stochastic gradient descent, were explored. One of these approaches is reducing the gradient variance through adaptive sampling to solve large-scale optimization's empirical risk minimization (ERM) problems. In this paper, we will explore how starting with a small sample and then geometrically increasing it and using the solution of the previous sample ERM to compute the new ERM. This will solve ERM problems with first-order optimization algorithms of sublinear convergence but with lower computational complexity. This paper starts with theoretical proof of the approach, followed by two experiments comparing the gradient descent with the adaptive sampling of the gradient descent and ADAM with adaptive sampling ADAM on different datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。