arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1901.07648math.OCcs.LGstat.ML
收藏

带SARAH的光滑和Optimization

Finite-Sum Smooth Optimization with SARAH

IBM Research(IBM研究院)

Lam M. Nguyen, Marten van Dijk, Dzung T. Phan, Phuong Ha Nguyen, Tsui-Wei Weng, Jayant R. Kalagnanam

摘要:

找到有限和平滑非凸目标函数$F(w)=\frac{1}{n} \sum_{i=1}^n f_i(w)$的一阶稳定点的随机一阶优化算法的总复杂度(测量为梯度计算的总数)已被证明至少为$Ω(\sqrt{n}/ε)$(对于$n \leq \mathcal{O}(ε^{-2})$),其中$ε$表示输出近似$\tilde{w}$的获得精度$\mathbb{E}[ \|\nabla F(\tilde{w})\|^2] \leq ε$(Fang等人,2018年)。在本文中,我们提供了一个收敛性分析的SARAH算法(Nguyen等人,2017 a;b)并实现与(Fang et al.,2018)当$n \leq \mathcal{O}(ε^{-2})$用于非凸问题时,直到常数因子。对于凸优化,我们提出了SARAH++与次线性收敛的一般凸和线性收敛的强凸问题,我们提供了一个实用的版本,各种数据集上的数值实验显示出更好的性能。

英文摘要:

The total complexity (measured as the total number of gradient computations) of a stochastic first-order optimization algorithm that finds a first-order stationary point of a finite-sum smooth nonconvex objective function $F(w)=\frac{1}{n} \sum_{i=1}^n f_i(w)$ has been proven to be at least $Ω(\sqrt{n}/ε)$ for $n \leq \mathcal{O}(ε^{-2})$ where $ε$ denotes the attained accuracy $\mathbb{E}[ \|\nabla F(\tilde{w})\|^2] \leq ε$ for the outputted approximation $\tilde{w}$ (Fang et al., 2018). In this paper, we provide a convergence analysis for a slightly modified version of the SARAH algorithm (Nguyen et al., 2017a;b) and achieve total complexity that matches the lower-bound worst case complexity in (Fang et al., 2018) up to a constant factor when $n \leq \mathcal{O}(ε^{-2})$ for nonconvex problems. For convex optimization, we propose SARAH++ with sublinear convergence for general convex and linear convergence for strongly convex problems; and we provide a practical version for which numerical experiments on various datasets show an improved performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑