arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1607.00559math.OCstat.ML
收藏

非均匀采样下的次采样Newton方法

Sub-sampled Newton Methods with Non-uniform Sampling

Peng Xu, Jiyan Yang, Farbod Roosta-Khorasani, Christopher Ré, Michael W. Mahoney

摘要:

我们考虑的问题,找到最小的凸函数$F: \mathbb R^d \rightarrow \mathbb R$的形式$F(w) := \sum_{i=1}^n f_i(w) + R(w)$的低秩分解的$\nabla^2 f_i(w)$是现成的。我们考虑的制度,其中$n \gg d$。作为二阶方法被证明是有效的,在寻找最小的高精度,在这项工作中,我们提出了随机牛顿型算法,利用\textit{non-uniform}子采样的$\{\nabla^2 f_i(w)\}_{i=1}^{n}$,以及不精确的更新,作为手段,以降低计算复杂性。为了捕获$\{\nabla^2 f_i(w)\}_{i=1}^{n}$中的重要项,考虑了基于{ \it块范数平方}和{ \it块部分杠杆分数}的两个非均匀采样分布。我们表明,在每次迭代时,非均匀采样在最$\mathcal O(d \log d)$项从$\{\nabla^2 f_i(w)\}_{i=1}^{n}$是足以实现线性平方收敛速率时,在$w$提供一个合适的初始点。此外,我们表明,我们的算法实现了较低的计算复杂度,表现出更强的鲁棒性和更好的依赖于问题的具体数量,如条件数,相比类似的现有方法,特别是那些基于均匀采样。最后,我们的经验表明,我们的方法至少是牛顿的方法与岭逻辑回归在几个真实的数据集的两倍。

英文摘要:

We consider the problem of finding the minimizer of a convex function $F: \mathbb R^d \rightarrow \mathbb R$ of the form $F(w) := \sum_{i=1}^n f_i(w) + R(w)$ where a low-rank factorization of $\nabla^2 f_i(w)$ is readily available. We consider the regime where $n \gg d$. As second-order methods prove to be effective in finding the minimizer to a high-precision, in this work, we propose randomized Newton-type algorithms that exploit \textit{non-uniform} sub-sampling of $\{\nabla^2 f_i(w)\}_{i=1}^{n}$, as well as inexact updates, as means to reduce the computational complexity. Two non-uniform sampling distributions based on {\it block norm squares} and {\it block partial leverage scores} are considered in order to capture important terms among $\{\nabla^2 f_i(w)\}_{i=1}^{n}$. We show that at each iteration non-uniformly sampling at most $\mathcal O(d \log d)$ terms from $\{\nabla^2 f_i(w)\}_{i=1}^{n}$ is sufficient to achieve a linear-quadratic convergence rate in $w$ when a suitable initial point is provided. In addition, we show that our algorithms achieve a lower computational complexity and exhibit more robustness and better dependence on problem specific quantities, such as the condition number, compared to similar existing methods, especially the ones based on uniform sampling. Finally, we empirically demonstrate that our methods are at least twice as fast as Newton's methods with ridge logistic regression on several real datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑