arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1412.6293math.NAcs.NAmath.OC
收藏

半随机坐标下降法

Semi-Stochastic Coordinate Descent

School of Mathematics(数学学院);University of Edinburgh(爱丁堡大学)

Jakub Konečný, Zheng Qu, Peter Richtárik

AI总结:

本文提出半随机坐标下降法(S2CD),用于最小化强凸函数的平均值。方法结合确定性步骤和随机步骤,通过非均匀分布选择函数和坐标,逐步改进梯度估计。

摘要:

本文提出了一种新颖的随机梯度方法---半随机坐标下降法(S2CD)---用于最小化强凸函数的平均值问题:$f(x)=\tfrac{1}{n}\sum_i f_i(x)$。我们的方法首先进行确定性步骤(计算函数$f$在起始点的梯度),随后进行大量随机步骤。该过程重复几次,最后的随机迭代点成为新的起始点,进行确定性步骤。该方法的创新之处在于随机步骤的执行方式。在每个随机步骤中,我们随机选择一个函数$f_i$和一个随机坐标$j$(均使用非均匀分布),并仅更新决策向量的一个坐标,基于计算函数$f_i$在两个不同点的第$j$个偏导数。每个随机步骤构成梯度的无偏估计,且步骤的平方范数在期望下趋于零,意味着梯度的随机估计逐步改进。该方法的复杂度由两个术语组成:$O(n\log(1/ε))$次梯度$\nabla f_i$的评估和$O(\hatκ\log(1/ε))$次偏导数$\nabla_j f_i$的评估,其中$\hatκ$是一个新的条件数。

英文摘要:

We propose a novel stochastic gradient method---semi-stochastic coordinate descent (S2CD)---for the problem of minimizing a strongly convex function represented as the average of a large number of smooth convex functions: $f(x)=\tfrac{1}{n}\sum_i f_i(x)$. Our method first performs a deterministic step (computation of the gradient of $f$ at the starting point), followed by a large number of stochastic steps. The process is repeated a few times, with the last stochastic iterate becoming the new starting point where the deterministic step is taken. The novelty of our method is in how the stochastic steps are performed. In each such step, we pick a random function $f_i$ and a random coordinate $j$---both using nonuniform distributions---and update a single coordinate of the decision vector only, based on the computation of the $j^{th}$ partial derivative of $f_i$ at two different points. Each random step of the method constitutes an unbiased estimate of the gradient of $f$ and moreover, the squared norm of the steps goes to zero in expectation, meaning that the stochastic estimate of the gradient progressively improves. The complexity of the method is the sum of two terms: $O(n\log(1/ε))$ evaluations of gradients $\nabla f_i$ and $O(\hatκ\log(1/ε))$ evaluations of partial derivatives $\nabla_j f_i$, where $\hatκ$ is a novel condition number.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑